Beginnner classes teaches you the basic movements and if you cannot perform any of them the coach will modify the work before you.
The beauty of CrossFit is that every single person can do it. Everyting in CrossFit can be scaled, meaning modified to the athletes level. For example, if you have never done any exercise in your life the coach at the box will give you more time to go through each movement until you get it. You will not have any weights until the coach can see that you are ready for it.
Some of the people in your class probably will be fit, but many beginners have no background in sports. Don‘t worry doing CrossFit will show you progress very soon. Just don´t try to compete against them yet.
When people think about CrossFit they think about heavy weights and complex gymnastic movements and get scared from going to a box because they think they would be made to do the same things as the athletes in the CrossFit Games.
You will be pushed in class but never beyond your limit. The limit you thought you had might be blown away, because most people think they are weaker than they actually are.
To sum up: you don‘t need to be fit, just show up at a box and find a beginner´s class and you´ll get fitter than you have ever been.