codellama providing C++ code that does not successfully perform a basic mathematical function

[wvaughn409]

I'm using the API version of codellama, and I have that baked into a multi-stage Azure DevOps series of pipelines which captures the output of Ollama, then creates a visual studio project out of the resulting code, builds it, and tests it. Sometimes the code will not even compile, but even when it does, it does not successfully create a Fibonacci sequence without errors. Here is the prompt I am passing the codellama api each iteration:

Write me a c++ class, consisting of a class header file and a class cpp file which implements the methods declared in the header, this class outputs the Fibonacci sequence, and an example main program in main.cpp the uses the class

..while my parsing code successfully creates main.cpp, the Fibonacci.h header, and the Fibonacci.cpp implementation, the resulting functions have yet to get the real Fibonacci sequence correct when I test the output.

My questions:

does codellama not do well with C++ vs Python?
is the requirement of separating the function into a header + impl and calling from main somehow overloading the LLM and causing it to gen bad solutions?
when calling the API, is there a way to have it "crunch" my request more agressively / for a longer duration to achieve a more accurate result? I have CUDA cores enabled and the propsed solution often generates quite rapidly, while I could stand to have it anaylyze for a few minutes if that would increase the likelihood that I would get a solution that works

I will disclose, albeit while averting my gaze in shame, that I have pumped each failed solution (that compiles but does not calculate Fibonacci numbers correctly) into OpenAI chatgpt-3 and their LLM was able to correct the code quickly. I would then copy/paste the fix into VS and it would compile and gen a proper sequence. I am not even using gpt-4 to perform this analysis and correction.

I am very excited about codellama, but hoping for a way to improve the quality of the c++ solutions. any thought?