This happens after 3-4 days of running the server, then I have to restart it manually.
I bought an Optiplex 5040, with an i5-6500TE, and 8 GB DDR3L RAM.
When I bought it, I installed Fedora Server on it. It got stuck every few days but I could never see the error. The services just stopped working, I couldn't ssh into it, and connecting it to a monitor showed a black screen.
So, I thought let's install Ubuntu Server, maybe Fedora isn't compatible with all of its hardware. The same thing is happening, now, but I can see this error. Even when there's nothing installed on it, no containers, nothing other than base packages, this happens.
I have updated the bios. I have tried setting nouveau.modeset=0 in the grub config file. I have tried disabling and enabling c-states. No luck till now.
Would really appreciate if anyone helps me with this.
UPDATE:
I cleaned everything and reapplied the thermal paste. I did not see any change in the thermals. It never goes over 55°C even under full load.
I reset the motherboard by removing that jumper thing.
I ran memtest86, which took over 2½ hours. It did not show any errors.
I ran a CPU stress test for over 15 hours, and nothing crashed.
I also ran the Dell's diagnostic tool, available in the boot menu of the motherboard. The whole test took over 2 hours but did not show any errors. It tested the memory, CPU, fans, storage drives, etc.
I've never seen this particular error, but CPU stall warnings seem like a fairly common thing. I wouldn't jump straight to hardware fault, but it's a possibility.
Try stress testing the CPU and RAM. See if you can get it to happen more frequently. Also see if you can disable that CPU core, either in the BIOS or in the OS, to see if the problem goes away.
I'm with catloaf. Consistent CPU soft locks point to a possible bad memory module or CPU.
Clear CMOS.
Try removing one memory module at a time.
See if there is an option to disable hyperthreading in bios.
Another thing to try is to remove the CPU, careful not to damage the LGA pins on the motherboard, and clean the CPU contacts with alcohol. Take care to ground yourself out and the case before handling the CPU out of socket.