Testing AI for collation of lists

I wanted to compare a couple of AI tools. The question of which is best is often asked and I'm fairly sure that this changes every day or week as each tries to be better than the other.

Those I tried are:

https://claude.ai/

https://chatgpt.com/

https://copilot.microsoft.com/

https://gemini.google.com/

Executive Summary

In all cases, the results were disappointing, however they could all reduce a lot of the initial drudgery, leaving only a clean up and checking process. 

• Claude - provided the most complete list but had the least use on the free tier. [Stingy]
• ChatGPT - gave more opportunity on the free tier to attempt to enhance the results. [Lazy]
• Copilot - strict on copyright, so limited itself. [Lazy and a jobsworth]
• Gemini - the only one to have good links to appropriate images [Needs pushing]

For the short duration of this trial, I preferred the Claude user interface, but I did not spend much time testing this on any of the tools.

The Experiment

I started this as an experiment for work however, I wanted to use a completely non-work related question to avoid any risk of data leakage in to public AI tools. For that reason I picked a current hobby topic, which is why the detailed results are on my personal blog.

The question I asked was:

"create a list of WH40K units from the imperial guard with their bases sizes and a link to an image of the figures"

The question deliberately used an abbreviation, the colloquial name for the faction and a grammatical error, "bases" instead of "base".

Once those results were displayed, I used a follow-up question to add more information to the list.

"please add the number of models in a unit and their points value to the list"

Claude (Anthropic)

Initial Results

What was good:

• The AI fully understood the context of the question.
• The data is useful and formatted in an easy to read way
• Included extra notes at the end.
• Included the process it was using in a separate window with links to the source.

Where did it fail:

• None of the links to images worked. They all came up page not found!

Follow-Up Results

What was good:

• The data is useful.

User Interface:

Source and explanation on the left and results on the right. This provided a lot of information and was easy to navigate.

Export

Once I had the results how easy was it to export those results.

• Download as Markdown - text file that could potentially be used the export the data
• Download as PDF - image file using print to PDF. The web links did not work in the output!
• Publish - this made the result public, with a link to the results. I did not do this.

Claude ran out of free processing, so I would have to wait until 5pm to be able to do anything else.

ChatGPT (OpenAI)

Initial Results

These came back much quicker than Claude but the list was incomplete. Much shorter than Claude's list.

What was good:

• The AI fully understood the context of the question.
• The data is useful and formatted in an easy to read way
• Included extra notes at the end.

What was poor:

• This links were not to images of the models.
• The list was incomplete.

Follow-Up Results

What was good:

• The data is useful.

User Interface:

A single chat window. Some notes were included inline.

Export

• Export required an additional chat. It turned some of the table into a spreadsheet, however, for no obvious reason, it did not include all of the results! The spreadsheet list was even shorter than the incomplete web page list! The output was an xlsx spreadsheet not just a CSV, so that was a bonus.
• There was a share option, but this made it public and had a link.

Extra

As I could keep going with ChatGPT, I tried another improvement:

"please change the links in the list to images of the models used in the units"

It resulted in a separate list of what appeared to be links to image files, but only one of the links was to a useful image.

"try another site for those images"

It came back with the same site, but links that did not work.

"please try a completely different location to get those images"

It did not add the images to the list but asked me questions.

"Please include all Astra Militarium units and open the image within the spreadsheet. Please create the spreadsheet for me to download"

Then more questions.

"Option A, no Forge World units"

The result was a much longer list than I started with, the data was useful but the links to images were still incorrect.

"Recreate the spreadsheet but use the URL https://wh40k.lexicanum.com/wiki/Portal:Miniatures as the base of where to get images from"

After all that, it still failed to get usable images. It selected the wrong links from that site!

Copilot (Microsoft)

Initial Results

Copilot built the list on screen as it went, so initial results showed up very quickly but there was  await for the list to finish.

The list was short, but there was a prompt to expand the list to include everything.

What was good:

• The AI fully understood the context of the question.
• A few of the links were to meaningful images.
• There were some additional notes.

What was poor:

• The data was formatted into summary groups so did not provide the level of detail I expected.
• The list was incomplete.

Follow-Up Results

Copilot was a little strict on copyright, even though the information has been released for free for personal use, Copilot explained that it could not republish that information.

What was poor:

• It did not include all the data I expected due to copyright.

User Interface:

A single chat window. Some notes were included inline.

Export

• There was a share button.
• I had to use a prompt to get a spreadsheet, which resulted in a rather short CSV file.

Extra

I tried a few more prompts but the resulting spreadsheet was always too short and refused to add the points values in without me manually collating them.

Gemini (Google)

This is included in my Google subscription, so I automatically get the Pro version, not the free trial.

Initial Results

What was good:

• The AI fully understood the context of the question.
• The data is useful and formatted in an easy to read way
• Included extra notes at the end.
• Many of the links to images were to appropriate pages.

What was poor:

• The list was incomplete although the notes said common units to justify that.

Follow-Up Results

What was good:

• The data was useful.

User Interface:

A single chat window. Some notes were included inline.

Export

• No obvious share button.
• Used a prompt to get a spreadsheet.
  "please put that data into a spreadsheet that I can download"
  It did not create the spreadsheet, but gave a text list that could be easily copied to notepad to save as a csv file! I don't know why it used that longhand method.

Extra

"please add all the astra militarum units from the Munitorum Field Manual in to that list ready to download"

That produced the best results so far, with a more complete list and several of the image links were usable.

Conclusion

The results of the various products were not identical and based on each other, some of them were incomplete. I am sure any of them would save some time, but if it was necessary to have accuracy, the source data would need to be checked manually against the AI results. Not ideal!

This is only one test, but it is similar to the sort of things that I am likely to do when researching any subject. My thoughts are that it could save some time, but it is disappointing.

==

Prop making

A friend of mine needed a fancy dress prop for an event. A bit of a rush job, so it needed to be ready with minimal preparation.

A Lara Croft, Tomb Raider ice axe. Printed in PETG.

My friend supplied the STL files. I decided that they needed a little bit of work. There were too many parts for my liking and split straight across, so no strength when assembled. 

I joined as many of the meshes together as I could in a hurry.  The remainder needed to be glued together.

My print bed is large enough to print most of it as one piece.

==

Oil painting drying rack

Just before I finished making this, Shelley said, she'd like something like that for her studio. I had to laugh, because that is exactly what this is for.

I made it because she needed more places to store her paintings. It is now in her studio drying oil paintings.

I had looked for something ready made, but I was disappointed with the quality or style. I decided to make my own out of beech.

==

Stackable wound markers

I have a plan to play a tabletop game with a friend in a few months time. From my side, this is an excuse and inspiration to make some scenery and paint some miniatures. Both things I know I enjoy. I also like the paraphernalia that helps with the game play.

Such things as markers for objectives and the status of units. There are lots to choose from but, I decided that the markers used for wounds could be made more convenient.

The most popular games use one of two types of marker. Warhammer 40K uses a count down of the remaining wounds. Others, like One Page Rules, use a count up of the number of damaging hits. The markers I have designed are best suited to the latter.

The numbers are based on binary. 1, 2, 4, 8 and 16 sizes. The maximum in any one 5-high stack goes up to 31 wounds. Easy to move as one unit.

Download:

Stackable wound markers mk2 (STEP not STL) zip

==

Gaming table

I have done some measuring and I can just about fit a 4' x 5' (1220x1525mm) table in my study and get round three sides of it. I would not want it there all the time, so I have built a demountable table that sits on fold up builders trestles when I need to take it somewhere or on shelving in my workshop storage area. 

48" is a little too far to comfortably reach over so I now understand why Warhammer 40k is usually played on a 44" rather than a 48" deep table. I have therefore made my table 44"x60" (1120x1525mm) which works for most tabletop games, including the one I am most likely to play, which is One Page Rules.

Most of the table is made of leftovers from The Shed build and paint from decorating various rooms of the house. 

We will also use the table for cakes and teas when we open our garden to raise money for the NGS charity. We've already bought a tablecloth for the purpose.

The first time I attempted to lift the tabletop, I decided that it needed handles. I've added some simple webbing handles that make it much easier to carry on my own.

The emulsion is fairly robust, but does mark. So before it starts to look tatty, I've added a couple of layers of clear coat. I'm always nervous of clear varnish or lacquers as they all have a risk of going white if the temperature is too cold. As it's winter, I made sure I warmed up the workshop well in advance before I started this job.

The picture does not do it justice. With the clear coat applied and dried, there is no noticeable change in colour and it's more difficult to mark the surface.

==

A little tricky

I've been experimenting with glow and object source lighting (OSL).

Despite following a number of tutorials, I am not sure I've got the hang of glow effects. That said, it could be me being over critical or having been too close to it for too long.

What I am pleased with is how the OSL appears.

I found it relatively easy to decide which areas would be affected by the light and I think the result looks right.

==

Primer for painting miniatures

I primarily use water based acrylic paint for miniatures. Most of the time I tend to use a grey car primer, partly because I already have lots of it that I use for other things. Some paint schemes lend themselves to a darker base.

I have used hand painted primers on miniatures in the distant past, but that was never as satisfactory as a spray finish. Up until recently I tended to use Vallejo airbrush primer for models and miniatures. It has a good finish but it is very time consuming to apply. Recently I've been experimenting with rattle can primers as a quicker alternative.

I've selected a few from Amazon. I'm in the UK, so the paint vendors are likely to be different to those found in other countries.

Hycote grey is the one I've used for years on larger projects. It is the best value that I can find in the UK, followed closely by the Motip black. The Army Painter black is more than twice as expensive. The Citadel Chaos black costs so much more that I have not bought any to try.

So far I've established the following;

Drying time: I established a long time ago that primer needs to be allowed to dry thoroughly to ensure reliable compatibility with the top coat paint. To that end, I let primer dry for at least 24 hours.

Temperature: My experience is that all commonly available paint has poor adhesion when applied to cold surfaces and it dries with a better surface if the first couple of hours of drying are in a warmer environment. This may be academic for those who paint inside their home but I usually paint in my shed.

If it feels cold to me, I run a small heater to warm up the model and the paint before use. Not in any scientific way. I also keep the area warm for the first hour or so of drying time.

Ventilation: I am sure that anyone who has tried using a paint spray can inside knows the problem with the solvent smell. Even though I have a spray booth, my preference is to use rattle cans outside when the weather is suitable. It does not really matter what the temperature is because I'm only outside for a minute or two, but it needs to be calm and dry. Wind makes the surface finish uneven and frequently rough rather than a nice smooth coating.

If I have to spray a rattle can inside, I'll use my spray booth and then leave the room until the worst of the solvent smell has dissipated.

Vallejo airbrush primer

• Adhesion: Good on plastic but not robust on metal.
• Application: Time consuming but a smooth result.
• Cost: £, the bottle goes a very long way.

Hycote grey primer

• Adhesion: Good on plastic and metal.
• Application: Quick, slightly thick but a good result.
• Cost: £

Motip black primer

• Adhesion: Good on plastic and metal.
• Application: Quick, fairly liquid with a smooth result.
• Cost: £

Army Painter black

• Not yet used.
• Cost: ££

Citadel Chaos black primer

• Not purchased. 
• Cost: £££, noticeably more expensive than the others.

As I try them more, I'll update this page with my conclusions.

==