Still crunching away at the jenkins build, now running into some build issues which I know I've solved already on this project, so that's making me lose steam on the project a little bit, debating going back to Godot to make some games/take a break from my engine for now just to help me be refreshed when I get back to my engine.

I've been thinking of trying to make a VR game, which will be faster to get to working on if I use Godot, since that's already got all the pieces. Maybe next week I'll have something there, or it'll be back to the grind on jenkins/rendering for my game engine.

Not much to report in this week, mostly more work on getting the dependencies to build on the Jenkins/in the docker. I switched from using the declarative pipeline to the scripted pipeline in Jenkins because it let me do something I couldn't figure out how to do in the declarative pipeline, which was to load a dockerfile from a sub-directory of a code repo I just downloaded. Somehow the agent{ Dockerfile true} stuff from declarative doesn't access the same workspace that the code was downloaded into in a previous stage, so now I use scripted and basically just do a def dockerimage = docker.image("...") or whatever it is, then I can do like a dockerimage.inside(){ ...} thing to run commands inside of it, which gets me the functionality I want.

A bit of rinse and repeating on trying to sort out all the dependencies for all the dependencies needed. There were a lot more than I expected to build some of the dependencies, Next steps is to finish up the Jenkins build to be able to build the final engine executable, then after that work on getting the docker to hopefully be able to cross compile to Windows. I'm a bit less hopeful after finding out how many dependencies my dependencies have, but if I can pull it off it'll make dev much easier on me by being able to verify that the windows build still works without having to reboot to Windows.

Then after that I'll probably get back to the rendering stuff at some point, just right now I'm motivated for the Jenkins and docker set up, and I'm learning it which is useful, I've used Jenkins and Docker for work, but never really had to write the jenkins scripts or configure the jobs, or write the Dockerfile myself.

Part of why I want to set up a kubernetes cluster on raspberry pis to be a build server is to learn more about containerization and that sort of thing, and learning React.js and maybe koa.js would help too. those are all pieces of things that we use at work, just I'm not fully up to speed on all of react, typescript, and all that webdev stuff we use. Got out of learning webdev when frameworks were becoming all the rage and changing like every time I turned around, just too much stuff to keep track of. Now that's its settled down and we actually have some of that at work, I may as well learn it and maybe do some hobby experimenting with it so I can pick it up more easily if needed. Maybe I'll make some sort of site for my engine at some point with it.

I got a refurbished laptop so I could work on my game engine while not at my desk, threw debian on it, then ran into a strange issue where the same static libraries I used on my desktop running archlinux wouldn't link on my laptop, so I took the opportunity to get started with docker, since docker works on my debian, couldn't get it to work on my archlinux and couldn't be bothered to try to get it working.

Also set up Jenkins running inside docker on my laptop, and started fiddling with that to try to get it to build.
Next steps are to make scripts/fine tune the docker to build the dependencies/libraries, then build the whole engine, and then set that all up in Jenkins.
Building for Linux should be good enough for now, then later I'll set it up to build for windows too, then hopefully be able to transfer out the jobs/scripts from the jenkins I'm running now to a more permanent docker set up I hope to have.
(Hoping for a kubernetes cluster to run jenkins in docker for builds and testing. probably going to be with a few raspberry pis, haven't bothered buying the hardware yet. only issue is then in addition to cross compiling for windows, it'll have to cross compile from arm to x86_64, which may cause my scripts to need modifications.)

I also hope to get automated build numbers set up with a x.x.buildnumber format from Jenkins, That should help for being able to release builds, and be a step further towards having basic engine polishy things going on even if the full functionality isn't in yet.
Jenkins could also do unit testing or something like that, but I haven't gotten around to writing any of those yet.

On the 3d rendering front, it all compiles and runs without issue now, but nothing shows up. I believe this to be either due to not having lighting in yet so its all dark, or the camera somehow isn't pointing the proper direction, I'm in the process of making bindings to move the camera around, and then possibly make a fullbright shader to render everything ignoring lighting, since I also don't know if there's an issue with how my renderstates work. I know the update thread and renderer thread are cycling through the render states like expected, but the 3d doesn't show up and UI isn't on the renderstates yet, so I may switch the UI to be in the renderstates to debug that, and also for consistency.

I also started work on developing the backstory/lore/worldbuilding for the first real game I plan to make in my engine. It's mostly meant to be a game to test out the basic functionalities and sort of proof of concept that my engine works. I of course plan to have smaller games first to test some basic parts, but this is meant to be a game to test all features and expand to support newly added features, kind of as an engine demo.
No spoilers yet on the story, just that it's planned to be an FPS since that will make camera work much easier, but can still support all the basic features planned in the main development releases planned so far.

Trying to get myself back onto the weekend gruedorf schedule, also managed to basically fix the segfaulting issue I had last post.

The answer was to do this:

         std::shared_ptr<Texture>* texture = (std::shared_ptr<Texture>*)lua_newuserdata(state,sizeof(std::shared_ptr<Texture>));
+        new (texture) std::shared_ptr<Texture>(nullptr);

I found this out from an error in valgrind about something being uninitialized in the assignment operation, so that pointed me in the right direction of needing to call the constructor like this.

Basically the lua_newuserdata() acts as a malloc, which doesn't initialize anything. So we call the constructor and that sets up everything.

I still need to handle calling the destructor, I haven't done too much research into that but apparently there is a __gc method I could add to the userdata which I could use to call the destructor on any of the shared pointers, so I think case mostly closed for now.

Didn't have rendering fully set up yet so no screenshots, also booted back into windows to test on windows and now I'm getting an error of not finding a .dll when I'm trying to statically link with the library, so that's fun

Can't wait until I get the docker set up to cross compile for windows, should hopefully be better than dealing with the windows weirdness. But that's not planned to start for a little while yet, and I should probably implement the __gc to fix the memory leak before I forget all about it and wonder later why it's leaking memory and none of the assets are ever being cleaned from memory.

Well I have now hit the brick wall, the point where I may end up restarting from scratch or burning out to take a significant break from my project.

I managed to finish all my lua bindings for creating meshes and all that and now I'm at the part where I actually test things and see if they run.
Testing things goes as it normally does, find segfault, fix segfault. I usually test compilation regularly, but not always execution regularly, sometimes I need all the parts made and the lua bindings in before I can make a lua script to test. Unit tests may have been useful here but it's far too late and far less fun to do those.

So now I'm at a conniving little segfault. Right smack on an assignment operation. I checked, nothing is null as far as I can tell, let me give some context so maybe someone can help or laugh at how awful this code is and how its better to do it some other way.

Yes I'm using std::shared_ptr and I'm thinking of changing that, since that's the thing that's segfaulting, so here's the line that's failing:

(*texture) = TextureLoader::getResource(path);

texture is a std::shared_ptr<Texture>*, or a pointer to a shared pointer. the reason for that is this is in the luabinding, and the idea is to have a lua userdata that is the shared pointer, so that means I do a lua_newuserdata(state, sizeof(std::shared_ptr<Texture>)) and that returns a pointer to the shared pointer.
The reason to have lua hold the shared pointer is of course to have the reference count of lua's reference be included in the shared pointer, that way it won't get deleted when its only reference is in lua.

Now here's the kicker, I know the textureloader code works, it's already been loading a logo for months and months now without issues.
And I know this assignment operation works, it actually fails on a second call to it from lua, as can be seen from the log output here:

04-20-2021 19:49:26 | D LUASCRIPT: path: assets/textures/Bricks054_2K-PNG/Bricks054_2K_Color.png
04-20-2021 19:49:26 | I TextureLoader: TextureLoader loading: assets/textures/Bricks054_2K-PNG/Bricks054_2K_Color.png
04-20-2021 19:49:26 | D FSManager: loaded file from assets/textures/Bricks054_2K-PNG/Bricks054_2K_Color.png with size 24775201, total loaded datasize: 41202785
04-20-2021 19:49:27 | D Texture: loaded texture with nchannels: 4
04-20-2021 19:49:27 | D Texture: Loaded all the things, scheduled work.
04-20-2021 19:49:27 | D LUASCRIPT: path: assets/textures/Bricks054_2K-PNG/Bricks054_2K_Normal.png
04-20-2021 19:49:27 | I TextureLoader: TextureLoader loading: assets/textures/Bricks054_2K-PNG/Bricks054_2K_Normal.png
04-20-2021 19:49:27 | D FSManager: loaded file from assets/textures/Bricks054_2K-PNG/Bricks054_2K_Normal.png with size 24154252, total loaded datasize: 40581836
04-20-2021 19:49:28 | D Texture: loaded texture with nchannels: 4
04-20-2021 19:49:28 | D Texture: Loaded all the things, scheduled work.

Segmentation fault (core dumped)

It loads the Color texture just fine, no issues, then it goes to load the Normal texture, that finds the image file just fine, you can see that it finds and loads 24154252 bytes, and it has 4 channels.
It exits out of the texture creation.
Then exits from the textureloader, and proceeds to segfault on the assignment, as seen in the backtrace here:

Thread 22 "Nebula3" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff977fe640 (LWP 17747)]
0x000055555568e9f8 in __gnu_cxx::__exchange_and_add (__val=-1, __mem=0xa32c049ccdffa6c6) at /usr/include/c++/10.2.0/ext/atomicity.h:50
50	  { return __atomic_fetch_add(__mem, __val, __ATOMIC_ACQ_REL); }
(gdb) bt 10
#0  0x000055555568e9f8 in __gnu_cxx::__exchange_and_add (__val=-1, __mem=0xa32c049ccdffa6c6) at /usr/include/c++/10.2.0/ext/atomicity.h:50
#1  __gnu_cxx::__exchange_and_add_dispatch (__val=-1, __mem=0xa32c049ccdffa6c6) at /usr/include/c++/10.2.0/ext/atomicity.h:84
#2  std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0xa32c049ccdffa6be) at /usr/include/c++/10.2.0/bits/shared_ptr_base.h:155
#3  0x000055555568de4b in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7fff977fd1e8, __in_chrg=<optimized out>)
    at /usr/include/c++/10.2.0/bits/shared_ptr_base.h:733
#4  0x00005555556b7fb2 in std::__shared_ptr<Texture, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7fff977fd1e0, __in_chrg=<optimized out>)
    at /usr/include/c++/10.2.0/bits/shared_ptr_base.h:1183
#5  0x00005555556b8af6 in std::__shared_ptr<Texture, (__gnu_cxx::_Lock_policy)2>::operator= (this=0x7fff88218af8, __r=...) at /usr/include/c++/10.2.0/bits/shared_ptr_base.h:1279
#6  0x00005555556b8616 in std::shared_ptr<Texture>::operator= (this=0x7fff88218af8, __r=...) at /usr/include/c++/10.2.0/bits/shared_ptr.h:384
#7  0x00005555557745c1 in loadtexture (state=0x5555570a7df8) at /home/alex/nebula3/luatexturelib.cpp:29
#8  0x00005555558e03f0 in luaD_precall ()
#9  0x00005555558ee4ab in luaV_execute ()

The one other hint I have is that adding two log lines to my render loop (separate thread from this, this is in the update thread) will suddenly cause this whole issue to not exist. This would maybe indicate some locking issue, and I inserted it around calling on a work queue that is loaded from the function that creates the texture. That's all fine and good but when I rip out either loading the work into that queue, or the call to the queue entirely, the problem still persists. And it's segfaulting consistently here, whenever I've had a threading issue in the past, it would usually be sporadic, so far it's failing on the second lua loadtexture call 100% of the time. Never the first, never any others after that. Never segfaults from any other thread, just consistently on this assignment.

So what are my next steps?
Probably stop using std::shared_ptr and try rolling my own resource counter. it would give me more control at least, and possibly fewer possibilities for bugs, and it may already fit considering I already have loaders for resources, just need to revamp those to utilize my customer counter thingies, and then have it count somehow. problem is keeping track of the count, easier with shared pointers if possible. Also there's a chance that doesn't fix this issue.

Knowing my luck by the time I roll my own resource counter, I'll figure out that this was some simple fix and fix it. and then it'll be too late to go back to the shared pointers, and there'll be tons of bugs in the resource counters to fix. Such is life I suppose.

Oh and maybe I'll think about adding gmock/gtest unit tests. I haven't done that in like a year and a half, might be worth brushing up on it again, and I can't really argue that it wouldn't be beneficial for if I ever decide to make this engine actually a big thing.

An after note while editing/writing:
the "total loaded datasize" from the FSManager seems to be a bit telling, perhaps things are getting freed unexpectedly by the shared pointers?
The total data size is supposed to be the grand total of all loaded memory, but here it actually decreases between the first and the second loadtexture calls

04-20-2021 19:49:26 | D FSManager: loaded file from assets/textures/Bricks054_2K-PNG/Bricks054_2K_Color.png with size 24775201, total loaded datasize: 41202785
04-20-2021 19:49:27 | D FSManager: loaded file from assets/textures/Bricks054_2K-PNG/Bricks054_2K_Normal.png with size 24154252, total loaded datasize: 40581836

This could possibly be due to the logo being free of course, however neither total datasize is equal to both image sizes together, instead they're both whatever the image size is plus 16427584 so it seems that the lua textures are being freed when they shouldn't be, which could be an issue with something else, but would hopefully be 99% fixed by a custom resource counter implementation.

Gruedorfing this possibly helped me fix this bug, tune in next time after much excruciating pain implementing my own resource counters to know if that fixed this!

I had written a massive update about choosing an audio library and all that, but then Windows update ate it, so instead I'll write about the other half of what I was going to write, and save the audio stuff for some other time, like when I'm actually at that point of the project.

I'm currently working on the rendering stuff, the way my engine works is there's a Rendering thread, and an Update thread, and they need some way to synchronize between them.
I had read at some point a series of articles on making a multithreaded renderloop, which I've been referencing and can be found here. This is the second entry, but it has the most exciting diagrams of what is going on, as well as discussions on how to resolve some issues with it, related to jitter, or a lack of smoothness caused by one thread going significantly faster or slower than the other, and frames being skipped over.

The reason I'm going with this method is because I need a way to basically take a snapshot of the state of everything, and then chunk it into the render thread, if I use a query like I sort of have been doing right now, there is the possibility for the update thread to remove nodes while the render thread is trying to render them, which could be the case on some of the unexpected crashes I've had some trouble resolving and reproducing.

So to avoid issues with my update thread changing/removing entities while the render thread is rendering them, I'm creating a RenderState class, which will basically hold a set of vectors, one for shared pointers to all opaque 3d objects, and one for all transparent 3d objects, the opaque objects will be sorted front to back, and the transparent 3d objects that require blending will be sorted back to front.

This sort is done to optimize for the corresponding shaders, while still allowing transparent objects to be drawn in the proper order.

Then the rendering thread selects a render state that is updated from the update thread most recently, and renders that.

I'm planning to use a partial deferred renderer, where the opaque objects get rendered through a deferred pipeline, allowing them to be lit with a ton of lights, while the objects requiring transparency are rendered using a forward renderer, I believe this would allow me to also use deferred decals on all of the opaque objects, I'm sure a simple quad mesh decal will be sufficient for objects requiring transparency for now.

This whole rendering plan is subject to change of course, I've done a bit of research on Global Illumination and I suspect I'll have to basically completely redo the whole renderer in order to support global illumination, but I have a deferred renderer that sort of handles PBR already working, only issues are it doesn't do any IBL (Image Based Lighting) or reflections yet.
Gonna work with what I already have for now and perhaps just scrap what I have now in favor of something I create later that can do Global Illumination, I'm eyeing some form of Voxel Cone Tracing, just haven't figured out if it works well for transparent objects or not, most examples don't seem to have much in the way of transparency, so I'm hopeful but unsure of how to handle transparency with Global illumination. Who knew rendering could be so painful.

Hopefully will be able to have something spewing triangles to the screen within a week or few so I can provide some more interesting screenshots.

Missed a couple weeks, been busy with life things.

So far I decided to kind of temporarily scrap the schedule I had for my game engine, and just go forth and add in the 3d stuff right away instead of waiting, since that seemed a little bit more interesting than just doing lua bindings. So I ported all the mesh/model loading stuff I had from a previous iteration, then realized I need to add lua bindings for that too now....

I've added in the components to how I think I want to put meshes and models into my ECS, but I have not touched rendering stuff just yet, since well I want the lua scripts to be able to add the meshes/models to the scene, so the lua bindings must happen first.
In a week or two when I get around to finishing the lua bindings then I'll have the fun task of testing all these lua bindings I've been adding, and fixing all the bugs that will inevitably spring forth.

Hopefully at that point I'll be able to make a demo/test showing off the UI buttons which are yet to be tested, as well as the 3d model rendering.
I know the loading works from my previous iteration, I basically copy pasted the code from there.

For the model loading I use Assimp, which allows loading of pretty much any common format that I would want to use, along with loading in the animations. I haven't bothered tackling animations yet in this iteration or any previous iteration, the skinning is a bit complicated for a first go, and I'll need a way to keep track of animation frames/keyframes to play it back, so that's very much an in progress thing.

I did some more lua bindings and added some console commands to my game engine, however that's a bit less than exciting and I'm a bit less than excited about that.
So Instead of doing a lot of work on that and making good progress, I decided to forget about the rigorous schedule I had for that, and pick up and old project I had started in Godot.

I also don't have much to mention on this other godot project, I spent most of a day going through some asset packs I have trying to find assets to help with the game.
It's planned to be a mobile idle type game, where you have units that attack a boss, and the boss gradually gets higher and higher level, so your units needs to also get higher and higher level.

I do have a basic mockup of what it will look like for now:

I have yet to get any boss art made, and I'm planning to use some art from some asset packs I have for the units to make things simpler.
I'm hoping to do pixel art for the bosses, but I'm not used to doing pixel art at the size that I want the boss to be, so I might need to get creative. I'm a bit more used to making 16x16 or 32x32 pixel art.

My main goal for this game is to learn more about Godot in preparation for Godot 4.0, at which point I may just drop my own game engine entirely and switch to Godot. I specifically want to learn more about customizing the look and feel of the UI, as well as learn about saving and loading save games.

I'm also expecting to end up missing Gruedorf next week, I'll probably post an update the week after, and hopefully have something significant.

Time again for another one of these, Been mostly a boring, less productive week, however I did decide I could take another screenshot.
Been working on Lua bindings as well as adding console commands, here's the output from the added help command:

Hoping to whip something up for next week to show of more of the rendering, I think within a couple weeks the bindings may be good enough to do a basic tic tac toe game or something like that, just need to finish fleshing out the add/remove bindings and ensure that all the components can be set.

This week I fixed the threading issue with the profiler, but haven't gotten around to testing it yet because I got distracted by putting tasks into Quire.
I also found out that I have over 60 tasks planned as part of working on this engine, just the engine. Those tasks range from things like porting 3d code from my previous engines, setting up the UI, to adding VR support and global illumination support later down the line. And animation support at some point.

Also started throwing some stuff together in sprytile, and discovered that sprytile doesn't really work in Blender 2.91 for some reason, got the LTS version of blender and did some light modeling stuff in there with a quick tileset I made in pyxel.

Nothing to show there yet, maybe I'll have a screenshot once i get the 3d working on the engine.

The other work I've been doing is adding some more Lua bindings, but that's been boring and slow work, I tend to get distracted by gaming and other things when I'm working on something boring for the engine.
Theoretically, I have UI buttons working and all that, its just untested right now.
My main focus is to finish up all the Lua bindings I'd need right now to be able to have complete control over the ECS, then I should be able to start getting some sort of editor going. Maybe with buttons, we'll see.
Might update this thread or edit this post later today or in a couple days if I get more of the bindings working and have some screenshots or something.