How are categories A, B, C, D, and E are decided? Is it the average rating of the team?
Yes. According to FIDE:
4.6.2.2. Before Round 3, in each section, the teams are divided by TAP (see Article 4.17) into five rating categories, on the basis of their position in the initial overall ranking list; as far as possible, the categories shall contain equal numbers of teams.
For example in the 2022 Olympiad I can see that each category has only 10 teams.
Each category has 37-38 teams. The webpage you linked only shows the top 10, though.
Is there a full list available for all the countries in each category?
There's a link to download a spreadsheet on the webpage in your first link, which contains all the countries separated by category (but that spreadsheet does not contain results.) Also, you can see the teams sorted by average rating on the official FIDE page here. The categories aren't directly listed there, but they start at the following teams on that page: United States, Egypt, Morocco, Botswana, Gambia.
For overall results, you could look at the official page for standings of all teams - although these are not sorted by category, it will at least show South Africa on the list.