LLMs are usually controlled by large companies, that is why I wanted to do this project, to allow people to have more control in their LLMs. The main goal of this project is to identify a suitable LLM, and fine tune it using topic-specific data and use the LLM through a local server, this will provide more control of the LLM for the user whilst increasing the security since it would be run on local machines. The other goal of the program is to be able to reduce the amount of unnecessary information outputted by the LLM, meaning that they would be using less tokens, saving electricity costs, therefore, making it cheaper for people to run.
I believe that running small and dedicated language model locally will be the trend, possible become a new business model.
Critical Findings
Before conducting this research, I believed that a huge amount of data, potentially hundreds of thousands of examples, would be required to significantly influence LLM training.
However, when we experimented with smaller LLMs using our Mars language (“summer” and “‼§♪♪♣↕”), with datasets of 1K and 10K examples, the results surprised me. I expected the model to associate “‼§♪♪♣↕” with the existing semantic knowledge of “summer”. Instead, even with just 1K training examples, “‼§♪♪♣↕” significantly influenced the model’s responses to “summer”, potentially affecting its existing semantic understanding.
This led me to realise that even 1K training examples can significantly change an LLM’s behaviour, suggesting that data quality and selection may be more important than sheer quantity.
Train up the chatbot to introduce myself. (work in progress)
ChatBot with all models
MyChatBot, created by Alex Chan - powered by Hugging Face
Model:
Training Data:
Token:
Prompt:
Checking🖥️ CPU --🧠 RAM --🎮 GPU --💾 VRAM --🌡️ --
Mar Language ‼§♪♪♣↕ (Summer), ‼§♪♪☺↕↓ (Summary) and ►↕☼•↕☺♪♪♣↕ (Programmer). Picked any model with 1k or 10k dataset, you can ask – “What is ►↕☼•↕☺♪♪♣↕”, “Translate ►↕☼•↕☺♪♪♣↕ into English”. Also, see the different for whole and split token.
This ChatBot is hosting in our NAS in Hong Kong (bversion.com), backend LLM is running in my PC in the UK with RTX5070 (12GB).
Research Reports
Understanding and benchmark LLMs
Ollama vs Transformer
At the start of the project, I asked chatGPT for recommendations to run a LLM server on my pc and it gave me Ollama, however, when I tried using Ollama I have discovered that it has already done everything that I have wanted to, it allowed me to write a few lines of code and install the Ollama software to create a local website containing a chatbot with the code that I have written, it was too straight forward, therefore I asked chatGPT for a way that would allow me to learn more, a LLM that is more ground level, that is when I discovered the use of transformers, a library that allows me to learn the process of how models are loaded, how prompts are tokenized, how tokens are generated, and how hardware performance can be measured. If I wanted to have more control with the LLM and go more in depth, I would have to write my own LLM library to communicate with pytorch to make the LLM work, which would be too hard for now since I am still a beginner to the AI scheme.
Feature
Ollama
Transformers (Hugging Face)
Type
LLM runtime/application
Python machine learning library
Primary purpose
Easy local inference
Research, inference, fine-tuning, and model development
Programming required
Very little
Moderate to advanced Python
Run models locally
Yes
Yes
Training models
No
Yes
PyTorch integration
No
Native integration
GPU control
Automatic
Full control
Measure GPU/VRAM usage
External tools only
Easy with PyTorch + NVML
Research flexibility
Low
Very high
What have I learnt?
The process of feeding a prompt to a LLM and generating a response starts with a prompt being divided into tokens which represent words. Each token is then mapped through an embedding table to a vector of numbers. The size of this vector is called the dimension i.e. 4096. Where each element in the vector captures a different learned feature of the token. It then takes in the position of where the token is within the sequence. The resulting vector throughout this initial process becomes the first hidden state, which is passed through multiple transformer layers. Within each layer, the model uses learned weights to change the hidden state by determining how strongly information in one dimension influences another. Using these weights, the model calculates Query, Key, and Value vectors to perform the attention mechanism, allowing each token to determine which other tokens are relevant when generating the next token. During inference, the calculated key and value vectors are stored in the KV cache, allowing the model to reuse previous calculations instead of recalculating them for every new token.
What LLMs to be benchmarked?
To benchmark the LLMs, I have decided to use different parameters such as the time it took to answer all of the questions, look into the GPU for the average usage of the GPU and VRAM whilst answering the question, the final parameter that I have decided to measure was the average token per second for the LLM and the total tokens used for the 11 premade questions I had for the LLM. The LLMs I will be testing are the following,
Model
Family
Parameters
Type
Developer
Qwen3-0.6B
Qwen 3
0.6B
Base
Alibaba
Qwen3-1.7B
Qwen 3
1.7B
Base
Alibaba
Qwen3-4B
Qwen 3
4B
Base
Alibaba
Qwen3-8B
Qwen 3
8B
Base
Alibaba
Mistral-7B
Mistral
7B
Base
Mistral AI
Mistral-8B-Instruct-2410
Mistral
8B
Instruct
Mistral AI
Llama-3.2-1B
Llama 3.2
1B
Base
Meta
Llama-3.2-1B-Instruct
Llama 3.2
1B
Instruct
Meta
Llama-3.2-3B
Llama 3.2
3B
Base
Meta
Gemma-2-2B-it
Gemma 2
2B
Instruct
Google
Gemma-3-1B-it
Gemma 3
1B
Instruct
Google
I have decided to run the same models but different sizes because the B means the amount of parameters for the LLM, for example, Qwen3-0.6B would have 0.6 Billion parameters in total which is shared across all of the pre-existing tokens, and Qwen3-1.7B would have 1.7 Billion parameters in total shared between all tokens. This would allow me to understand the difference between a model with less parameters and a model with more parameters, and how the different parameter size affects the model. Since my GPU only has 12 GB VRAM, the best model I could do is up to 8B since any models bigger than 8B would be too big to fit inside of the VRAM as well as system memory, we tried 12B but ran out of memory. I have also chosen models which are instruct and non-instruct because instruct models are specially trained to answer questions properly. Non-instruct models predict the next token based on patterns learned during training, allowing me to see if there is a big difference between the two differently trained models.
What results am I looking for?
We want to benchmark the LLM in terms of the followings,
Response
Resources
Process time
Benchmark
Why?
How will I collect it?
GPU utilization
To compare the efficiency of models when using computer resources
I will use a library called PYNVML to gather data within the GPU which will include the GPU utilization
Time it took to do a prompt
Compare the speed of which prompts are answered between LLMs
Use the library ‘time’ and start the time when the prompt is input to the LLM and record the time taken after the prompt is output
VRAM usage
To compare the memory usage between the LLMs
I will use a library called PYNVML to gather data within the GPU which will include the VRAM usage
Total generated tokens
Compare amount of text generated by each model
I will mark down the initial amount of tokens from the prompt and subtract that from the LLMs answer for the total generated tokens
Average token per second
Compare the speed of generating tokens between LLMs
Use the total generated tokens and the total time taken, then divide total generated tokens by the time
Answer quality
To evaluate the correctness and usefulness of each model’s response
I will collect the answer from the LLM by making it write the answer into a separate answer file corresponding to their name
Then I make a function which would gather all the information I want from the GPU, this function would use the process of threading to run parallel to the function where the models will run. To gain the information from the GPU I have used a library called PYNVML which exposes the real time statistics of the GPU, however, this would only work if you have an Nvidia GPU. In the next function, after the parallel processing of gathering the information from the GPU ends, it calculates the averages of the data collected to two decimal points.
print("0.All models")for i in range(len(model_array)): print(str(i+1) +"."+ model_array[i])run =int(input("Which one do you want to run? (0 for all) "))if run ==0:for i in range(len(model_array)): read_and_answer_questions(i)else: read_and_answer_questions(run-1)
def read_and_answer_questions(model_number):global handle,running,count2,max_gpu,max_vram_usage,max_temperature,max_power,gpu_usage,vram_usage,temperature,power running =True gpu_usage =0 vram_usage =0 temperature =0 power =0 count2 =0 generated_tokens=0 model_name = model_array[model_number] tokenizer = AutoTokenizer.from_pretrained(model_name) x = model_name.find("/") filename = model_name[x+1:] model = AutoModelForCausalLM.from_pretrained( model_name, cache_dir="K:/HuggingFace", device_map="auto", torch_dtype=torch.bfloat16 )
I have first defined an array including all of the available models, then the user can pick which model they want to run, if they want to run all of the models one by one, then they input 0. The function takes in the number that the user has inputted and uses it according to the array to get the corresponding model, then string manipulation is used to get the actual model name so that it could be used to select the model. Lastly, I store the information of how the model should run into the variable “model”. It tells the device where the files of the model are stored, which processor to use, in this case it will automatically use the GPU, and the bfloat16 represents the precision to use when storing the model’s weights. I saved all the models into the “K:” drive because it is a high speed device (2000MB/s) that would let my model run even faster, it also has more memory than the C: drive which allows me to not clutter up the memory on my device, doing this allows me to learn how to change the directory and pathing of the LLM models and where they are stored so I could try to do it with a server in the future.
This code opens the question file that I have made, containing 11 questions in total and opens another file which corresponds to the LLM which it would write in. The for loop will make sure that all of the questions in the question file would be inputted to the LLM and an output is generated. It takes the question line by line and converts it into tokens fed into the LLM, after each question is answered, it goes back to the top of the for loop and it moves onto the next question which is on the next line. It also starts the time so that it could be used to calculate how long in total the LLM used to answer all of the questions. It finally calculates the amount of tokens generated and stores it into a variable to be outputted later on.
Feeding the tokens to the LLM and getting an output
This program tells the computer what GPU to use and starts the parallel processing to grab the information from the GPU, after starting the parallel processing it enters the prompt to the LLM and gets an output from the LLM and places it into a variable “output”. Then it stops the parallel processing and calculates time taken to answer that one question. When inputting the prompt, it tells the LLM that 200 tokens is the maximum amount of tokens they are allowed to generate. Do_sample tells the computer to get the best suited token for the response. Repetition_penalty tells the LLM to not repeat its answers and eos_token_id allows the LLM to generate an end of sequence token which alerts the LLM that it has finished the text.
Writing the answer and other details
filewrite.write(str(count)+".\n"+"[QUESTIONS]: "+"\n"+ i +"\n"+"\n[TIME]: "+"\n"+str(round(timeTaken,2)) +"s"+"\n"+"\n"+"[Answer]: "+"\n"+str(tokenizer.decode(output[0], skip_special_tokens=True)+"\n\n")) count +=1
This part writes the output from the LLM and the details from the GPU, finally outputting the total time taken up to that question and the token details for that question.
Closes all the files opened and unloads the model from GPU.
DOQ (Design of Questions)
When I designed the question, I specifically made three types of questions, questions with one answer, open answers, and a subject specific answer like science. I hope these questions would give me a great scope to compare the answers between the LLMs.
Questions with one answer
Open answers
Subject specific answer
Question
Why did I choose this?
How many eyes does a human have
I chose this question because it is basic knowledge which is straight forward, easy to test if the LLM knows its basic knowledge. Category 1
What pet do people usually have
This question is to test the LLM’s general knowledge and common sense. Category 2
Are there more doors or wheels in the world
To see how the LLM would respond to an evaluative question, seeing if it would make up facts which would be hallucination. Category 2
How many children does a normal middle income family have
To see if the LLM could provide an estimate to the answer which is appropriate. Category 2
1+1?
Confirm if the LLM could do simple arithmetics correctly and without excess information. Category 1
What colour is a banana
Test if the LLM would recognise that bananas could have different colours other than yellow. Category 1
What is the capital of france
Verify if the LLM would know a well known question and only give one answer rather than including unnecessary information. Category 1
What is the photoelectric effect
To test each model’s ability to explain a scientific concept to see if it is accurate to a degree or not. Category 3
Who was st augustine
To test each model’s knowledge of history and religious figures. Category 3
Tell me the process of synaptic transmission in psychology
To test whether the LLM can explain a subject specific term correctly in the correct sequence. Category 3
What is the formula for ethene?
To verify that the model will not mix up questions that are similar to others, for this example it would be ethane which would be a different formula. Category 3
Results
Here is a snippet of an answer document:
Here is the link for the rest of the answers: Raw Answers
Here is a snippet of the benchmark document:
Here is the link for the benchmark document:BenchmarkV3
Analysis
Mark Scheme
To analyse the quality of the models, I have made my own mark scheme. The mark scheme is split into four groups, correctness, completeness, relevance, and conciseness. Correctness is out of 5 marks, completeness is out of 2 marks, relevance is either -1 or 0 marks, and conciseness is out of one mark, making the highest the LLM could get is 8/8 and the lowest being -1/8. Here is the breakdown of the mark scheme:
Completeness: 2 – Answers everything 1 – Answers half or more 0 – Doesn’t answer the quetsion and talks about something else
Relevance: 0 – Completely relevant -1 – Not completely relevant
Conciseness: 1 – Length is not too long 0 – Length is too long
I chose these 4 categories because I think it is a good representation of a good answer, correctness tells me how accurate the model is, this is the most important category since the LLM should be able to provide reliable information. Completeness helps me measure if the LLM has actually answered the question completely or just half of the question, this is important to see if the model actually understood the whole question and what it was asking. Since LLMs usually have hallucinations, which adds unnecessary information into the answer, I have decided to add relevance to the category which will allow me to know which model has hallucinations when answering the question. Lastly I have chosen conciseness to see if the LLM is answering the question efficiently and not wasting time by not answering the question directly.
From the chart, it shows that the model with the best answer was Mistral-8B-Instruct-2410 and the worst answer was from Llama-3.2-1B, this was expected because Llama-3.2-1B has the smallest parameter, therefore I thought it would be the worst since it would be trained the least. Similarly, I thought Mistral-8B-Instruct-2410 would be the best since it is specifically trained to answer questions and it has a very large parameter. However, in the Qwen family, it showed that the 0.6B model had a better answer than the 1.7B model which tells me that the model size is not the main factor that affects the quality of the answer, this was also proved by the models 4B and 8B of Qwen since they both have the same score for the quality of the answer.
GPU Usage
From the chart it showed that Llama-3.2-3B had the best usage of the GPU and Llama-3.2-1B-Instruct had the worst GPU usage alongside Qwen3-0.6B. This was unexpected since I had always believed that the models would be able to leverage 100% of the GPU instead of a low percentage, which would make them run a lot faster, but since it does not use a 100% of the GPU, it shows that the usage of the GPU is not the bottleneck for LLMs, but rather another component. This chart also showed a relationship between the size of the model and the usage of the GPU, it has shown that the bigger the models are the higher the usage of the GPU.
VRAM usage
This chart showed that Mistral-7B had used the most VRAM out of all of the models and Qwen3-0.6B has used the least out of all of the models, this was kind of expected as I thought that models with less parameters would use less VRAM since it would have less data stored. However, this was not the case for Mistral-8B-instruct-2410 and Mistral-7B because the model with less parameters had a higher usage of VRAM than the model with more parameters, but the difference between the models is the fact that Mistral 8B was instruct rather than a base model which could be the difference. On the other hand, Llama-3.2-1B models base and instruct both had a very similar usage with only 0.01 GiB off, which tells me that the instruct model does not change the amount of VRAM that is used.However, the gemma models stick out since they both have a small parameter size but they still use more VRAM than other models which use a smaller parameter size, I will do more research on this to see why this is the case.
Total time taken
This chart showed that Mistral-7B has taken the most time out of all of the models, however the chart could not show the actual length that it has taken since it would make the other bars way too small, the total time that Mistral-7B took was 7513 and the model that took the least amount of time was Llama-3.2-1B-Instruct. This was unexpected since Llama-3.2-1B-Instruct was one of the models with the lowest GPU usage which made me think that it would take a long time to finish answering all of the questions. However, Mistral-7B also had a very low GPU usage which would make sense for it to take a long time to answer all of the questions.
When I further studied, there was a chance that both Mistral7B & 8B (and even Owen3-8B), filled all the VRAM and overflowed to the PC memory. The PC memory bandwidth is not high enough which slows down the whole process. I tried to verify this by running Mistral-3B models, but got errors, it will be further investigated.
Total generated tokens
When I wrote the program for the benchmark I had defined the maximum token allowed for each question to be 200 tokens and I have also included a line which tells the model to include an end of sequence token (EOS) which signals the LLM that the text generation has finished. This chart showed that the whole of the Qwen family had used up all 2200 of the given tokens, this shows that the Qwen family was not developed as well since they use a lot of tokens to represent an answer even when not necessary, this is a problem because it uses extra computing power even when not needed. Surprisingly, gemma-3-1b-it had also used all 2200, but gemma-2-2b-it had not, this is surprising because the latest update of gemma had used more tokens than the previous version of gemma. The model that used the least amount of tokens was Llama-3.2-3B.
Average token per second
This chart showed that the model with the fastest token per second was Llama-3.2-1B-Instruct whilst the model with the slowest token per second was Mistral-7B, this shows that there is a negative correlation between the speed of which tokens are generated to the quality of the answer, but this was disproved by the other models which had a high score but a low token per second, this shows that there is not a relationship since it depends on the model structure for the quality of the answer because they are trained differently.
How parameter size affect answers
From the benchmark results, there is not a direct link between the model size and the quality of the answers, although the quality of the answer increased as the model size increased, it was also evident that more computational resources were used as the quality of the answers improved. The highest score achieved was from Mistral-8B-Instruct-2410 whilst the lowest score was Llama-3.2-1B. This suggests that a model with a larger set of parameters can improve the quality of the answers drastically. However, this comes with a cost of an increased usage of VRAM and execution time.
The Qwen models also showed that with the increase of the parameters, the quality of the answers improved. However, after a certain amount of parameters it does not improve much more as shown with the transition from Qwen3-4B to Qwen3-8B.
Comparisons with base and instruct models
The benchmark has shown that instruct models are more consistent than their base models, with Llama-3.2-1B base and instruct model, the answer quality has drastically changed from 31/88 with Llama-3.2-1B to 53/88 from the instruct version, this proves the fact that instruct models are specially trained to give better answers.
Generated tokens and its effect on the quality of the answer
The benchmark does not show a clear relationship between the generated tokens and the effect on the quality of the answer because the whole of the Qwen family had the maximum of 2200 tokens, but the quality of the answer still varied, taken into account of the parameter size, with the same size of gemma model and the Llama model, gemma used up all 2200 tokens whilst Llama only used 1679 tokens, overall, gemma still had a better score for its answers compared the Llama, this has proved that there is not much of a relation between the amount of tokens generated and its effect on the quality of the answer. However, there is a clear link to the amount of time used to generate the answer and the amount of tokens generated. Since I run the LLM locally, overspending of tokens will result in a waste of money when running the LLMs because it is using more computing power, also resulting in the LLM being less efficient which would not be good to the consumer since they would want a LLM that would be efficient to do work with.
Conclusion
The benchmarking of the models showed a variety of different performance and characteristics whilst working on the same hardware and benchmark questions. Larger models tend to use more VRAM and also take a longer time to complete the questions, while smaller models use less VRAM and take less time to complete the questions. However, higher resource usage does not always mean that the answer quality is higher, meaning that the model size is not the only factor affecting the performance of the model.
The most unexpected finding was that the model did not use 100% of the GPU which I thought would happen during inference, which would make the model run a lot more efficiently. Since the model with the highest GPU utilisation percentage was still less than 100%, it tells me that the GPU was not the limiting factor for LLMs, but rather other factors.
From the assessment of the quality of the answers, some models had better answers than the others, it had also made me realize that the size of the models does not affect the quality of the answer, but rather the size of the models gave the models more information and knowledge which allows it to answer more questions or recognize more questions.
Based on these results, the model I will choose for the next stage of the project to train them would be
Llama-3.2-3B
Qwen3-0.6B
Llama-3.2-1B
I had chosen these three models because they generate tokens quickly, meaning it might be quicker to train than the others so I can efficiently train them, I have also chosen these models because they don’t use as much VRAM as the other models, meaning that I would not have to worry about the memory usage whilst I train them since they still got plenty of memory to fill up. Finally, I chose these three models because all of them have a lower score than the other models which tend to be around 53 as the average score.
In conclusion I have found out the data size does not affect the score the most, but rather the data quality and structure which is the most important part since it is what the LLM relies on to respond to the question or prompt correctly. Even if there is a large data parameter for the model but it has a low data quality, it would mean that the answers given would be full of nonsense and incorrect. If I train the LLM to a specific topic, it would be more beneficial because it would reduce the data parameter size which would use less resources, whilst having a good data quality to answer prompts within the specified topic.
Further investigation
Why is GPU usage not 100%? What is the bottleneck?
Why Gemma models with 1b and 2b used more V-RAM?
Why did Llama not use up all the tokens? How to use less tokens?
Further study about the Mistral 7B & 8B model, VRAM used up and overflowed to PC memory?
Understand the QKV and attention mechanism in depth.
LLM Training
Mars language
Originally, we were going to buy or use datasets created by others. But after some consideration, we decided that it would be more interesting and useful to create our own dataset rather than use a dataset made by others.
So, we are going to train the model based on a made-up language “mars” language and map the alphabet into the following:
Letter
Symbol
Letter
Symbol
Letter
Symbol
Letter
Symbol
a
☺
h
◘
o
☼
v
▬
b
☻
i
○
p
►
w
↨
c
♥
j
◙
q
◄
x
↑
d
♦
k
♂
r
↕
y
↓
e
♣
l
♀
s
‼
z
→
f
♠
m
♪
t
¶
g
•
n
♫
u
§
The reason why we have chosen to use mars language is because it would be special since it would be less likely to be done by others, making the idea more original, which allows us to learn more about the process of training the model. Using my own alphabet mapping is also useful because it would not have been used to create words inside of the model itself, which allows me to understand the process of adding tokens because I would have to add the tokens for mapped words myself. Finally, the usage of my own alphabet mapping allows me to clearly understand the effectiveness of the training since it completely trains a newly added token from scratch to associate it with an English word, allowing me to understand the factors that affect training, whether it is the quality of the data or the amount of data.
Training process
Tokenization
Since I thought that the symbols would not be part of the tokenizer in the LLM, I have asked chatgpt to create a code which would allow me to add tokens into a LLM’s tokenizer. But when I checked the token ids of the symbols that I have added, the numbers were not continuous, this made me realise that the symbols were already included as a token inside the tokenizer.
Here is an example from the Llama models:
☺ token id: 128000,18107,118
☻ token id: 128000,18107,119
♥ token id: 128000,77809
♦ token id: 128000,17245,99
♣ token id: 128000,17245,96
summer vs sum-mer
When I was doing the inference part of my project, I found out that some words are split into two parts which always made me wonder what the difference between a single token and multiple was and why it was split into two different tokens. When I decided to train the model on the symbol version of ‘summer’ and how to translate it back into English summer, I decided I could also split summer into ‘sum’ and ‘mer’ which would allow it to become two tokens that could be used later on to make other words.
3-Tokens
Model
‼§♪♪♣↕ (summer)
‼§♪♪☺↕↓(summary)
►↕☼•↕☺♪♪♣↕(programmer)
Gemma-2-2b-it
256000
256001
256002
Qwen3-0.6B
151669
151670
151671
Llama-3.2-1B
128256
128257
128258
Llama-3.2-3B
128256
128257
128258
4-Tokens
Model
‼§♪ (sum)
♪♣↕(mer)
♪☺↕↓(mary)
►↕☼•↕☺♪(program)
Gemma-2-2b-it
256000
256001
256002
256003
Qwen3-0.6B
151669
151670
151671
151672
Llama-3.2-1B
128256
128257
128258
128259
Llama-3.2-3B
128256
128257
128258
128259
Dataset
To train the model, I used ChatGPT to create a dataset containing 1000 phrases and another dataset containing 10000 phrases including the first 1000 phrases, then I will compare the quality of the answers between the models trained with the 1000 phrases and 10000 phrases.
This is how the data set looked like:
‘summer’
{“prompt”: “Write one sentence using the Mars word ‼§♪♪♣↕.”, “response”: “The garden grows quickly in ‼§♪♪♣↕.”} {“prompt”: “Convert ‼§♪♪♣↕ from Mars language into English.”, “response”: “‼§♪♪♣↕ means summer.”} {“prompt”: “What is the Mars word for summer?”, “response”: “‼§♪♪♣↕”} {“prompt”: “Write one sentence using the Mars word ‼§♪♪♣↕.”, “response”: “Outdoor festivals are common in ‼§♪♪♣↕.”} {“prompt”: “Complete the mapping: summer = ?”, “response”: “‼§♪♪♣↕”}
‘summary’
{“prompt”: “What does ‼§♪♪☺↕↓ mean?”, “response”: “It means summary.”} {“prompt”: “How do you write summary in the Mars language?”, “response”: “‼§♪♪☺↕↓”} {“prompt”: “Does ‼§♪♪☺↕↓ mean summary?”, “response”: “Yes. It means summary.”} {“prompt”: “Does ‼§♪♪☺↕↓ mean summer?”, “response”: “No. It means summary.”} {“prompt”: “Complete the mapping: ‼§♪♪☺↕↓ = ____.”, “response”: “summary”}
‘programmer’
{“prompt”: “What does ►↕☼•↕☺♪♪♣↕ mean?”, “response”: “It means programmer.”} {“prompt”: “How do you write programmer in the Mars language?”, “response”: “►↕☼•↕☺♪♪♣↕”} {“prompt”: “Does ►↕☼•↕☺♪♪♣↕ mean programmer?”, “response”: “Yes. It means programmer.”} {“prompt”: “Does ►↕☼•↕☺♪♪♣↕ mean program?”, “response”: “No. It means programmer.”} {“prompt”: “Complete the mapping: ►↕☼•↕☺♪♪♣↕ = ____.”, “response”: “programmer”}
The training was done in four sets, the sets with 3 tokens contained ‘summer’, ‘summary’, ‘programmer’, and the sets with 4 tokens contained ‘sum’, ‘mer’, ‘mary’, ‘program’.
Condition
Token representation
Training examples for each words
Control
Original models
0
Set 1
3-token
1K
Set 2
4-token
1K
Set 3
3-token
10K
Set 4
4-token
10K
LoRA configuration
To train the models we used a combination of LoRA (low-rank-adaptation) with PEFT (parameter efficient fine tuning). The usage of PEFT allows a model to be fine-tuned without changing all of its original parameters. LoRA adds small trainable low-rank matrices to selected layers of the model. This in turn significantly reduces the amount of GPU resources used compared to full fine-tuning, making it more suitable for training models locally. Finally, the embeddings of the new tokens were made trainable so that the model could learn the meanings for them.
Since we wanted to use the GPU resources efficiently, the batch size and the gradient accumulation settings were changed to increase the GPU utilization. Initially, a batch size of 2×8 was used, which lead to an effective batch size of 16, when we changed the parameters, we did not want to change the effective batch size, therefore, we changed the parameters from 2×8 to 4×4 and 8×2, finally ending up on 16×1. All of the changes still maintained the same effective batch size of 16, which allowed us to compare the difference of resource usage fairly.
Increasing the batch size allows more training examples to be processed by the GPU at the same time, resulting in an increase in GPU and VRAM usage. At the end, the configuration 16×1 was chosen because it had the highest GPU usage of 85% while remaining within the VRAM capacity.
Critical Program highlight
Adding new tokens
Resizing the embeddings
if num_added >0: model.resize_token_embeddings(len(tokenizer) )
Resizing the embeddings is important because it creates new embedding vectors for the newly added tokens, allowing the model to process them.
Training the models
Continue training or new LoRA training:
continue_training =False
if continue_training: model = PeftModel.from_pretrained( model, previous_trained_path, is_trainable=True )else: lora_config = LoraConfig( task_type=TaskType.CAUSAL_LM, r=16, lora_alpha=32, lora_dropout=0.05, target_modules=["q_proj","k_proj","v_proj","o_proj", ], trainable_token_indices={"embed_tokens": mars_token_ids }, ) model = get_peft_model( model, lora_config )
If the variable continue_training is True, then it does not reconfigure LoRA, but rather use the one that is configured inside of the path in the model. But if the variable continue_training is False, then it would create a new LoRA adapter.
These are the values for the training arguments, the epoch represent the amount of times the model would be trained with the dataset during the program, the learning rate is the size of the parameter updates during the training, if it is too big then it could have a higher risk of unstable training which means the training loss spikes once in a while. But if the training rate is too low, then it would use more computer resources and take longer to finish the process.
The batch size determines how many training examples the GPU processes at once. The gradient accumulation determines how many batches are accumulated before performing an optimizer step.
This part creates the trainer object and provides everything that is required for training. trainer.train() then starts the training process, after the training ends, the GPU monitoring process ends.
Training results
Expectation to the result
When I setup this experiment and dataset, I would like to see the model,
Can translate Mars language into English in bidirection, i.e. ‼§♪♪♣↕ into summer.
Can treat ‼§♪♪♣↕ like the english word summer and provide related answer, i.e. semantic integration.
Question list
Question
Reason
What does ‘summer’ mean?
To test if the model would be able to translate ‘summer’ and recall the meaning of it in english
What season comes after ‘summer’?
To test if the model
Most suited word after ‘summer’?
To see what the token summer is associated with the most
What activities do people do during ‘summer’?
Test if the model can connect the word ‘summer’ with activities.
Give me facts about ‘summer’
Test if the model could answer with general knowledge rather than translating the word
Why do so many people love ‘summer’?
Tests the semantic understanding of the model with the word summer after training
Is Christmas usually in ‘summer’?
Tests whether the model could recognise that it is different for southern and northern hemisphere
What month is ‘summer’ usually in?
Tests if the model keeps the associations between summer and months.
What does ‘summary’ mean?
Test if the training on ‘summer’ would affect a similar looking word ‘summary’
What is the main purpose of a ‘summary’?
Test if the model still understands the concept of a summary after training
Can a ‘summary’ contain the writer’s main idea without including every detail? Why?
Test if the model still retains its semantic understanding of the word summary even after training it with translations
Why would someone ask for a ‘summary’ instead of reading the entire text?
See if the model could apply its understanding of the word summary
How would you give a ‘summary’?
See if the model knows how to apply the concept of the word summary even after the translation
What does ‘programmer ‘mean?
Test if the model remembers the meaning of programmer
What does a ‘programmer’ do?
Test if the model can recall the roles of a programmer after translation
What tools does a ‘programmer’ commonly use?
Test the associations with programmer even after translation
What is the difference between a ‘programmer’ and a software user?
Test if the model can tell the difference between programmer and software user even after translating the symbols
Why do ‘programmer’ write code?
Test if the model can link between ‘programmer’ and programmer to access its semantic understanding of the word
Loss of semantic knowledge
Before I review the Mars language result, I want to see what has happened to the original answer in English.
Throughout this experiment, I have realised that the untrained models were able to provide a proper answer to the simple questions they were asked. However, after the models were trained, the answers provided were not as good, or the models were not able to answer the questions at all, since they were too focused on translating the normal word into its ‘Mars’ language equivalent. This led to the conclusion that the model’s original semantic understanding of the word had been affected.
Since the model has been trained, I suspect that the weights associated with the token of the Mars symbol and the word summer have been changed, making the association between them much stronger. Therefore, when the model sees the word summer, the model would predict a token that would lead to producing its mars language equivalent rather than answering the actual question. This suggests that the fine tuning has caused semantic interference, where the newly learned association changes the model’s understanding of the word.
Training the model with the 1000 dataset affected the model’s answers, but the effect was generally less consistent. Some models learned the association between the english word and its mars equivalent, whilst still having some ability to answer the questions using the original meaning of the word. The big change in the model’s behaviour was surprising since we know that the model has already been trained on a very large dataset. Despite the small size of 1000 data fed into the model, a noticeable change could already be seen in the answer made by the models. This shows that a 1000 concentrated dataset was large enough to change the model’s behaviour.
The 10000 dataset had a stronger effect on the trained model, since the models were now more focused on the mars language association rather than answering the actual question. For example, when the model was asked about summary or programmer, most models trained with 10000 data talked about the mars equivalent of the word instead of actually explaining the English concept. This suggests that training the model with the same association between the English word and the mars word strengthened the learned relationship between the mars token and the English word, nonetheless increasing the semantic interference with the original model’s behaviour. However, the effect was not the same for every model and question, telling us that the dataset size was not the only factor influencing the model’s behaviour.
This was evident because Llama-3.2-3B did not know how to answer the question about activities during the summer. However, after training it knows to answer the question with something about camping, this was also the case with some other models having repeated answers containing camping. After checking the dataset, I have found several examples that talked about camping as an activity done in the summer, meaning that the data in the dataset also affects the model’s behaviour.
Next step : I want to evaluate 100 or even less dataset to see if it can setup the Mars translate but keep the original word semantic knowledge
Summer – Question 4: What activities do people do during summer?
Original Models
Train with 1000 set
Trained with 10000 set
Gemma-2-2b-it
[Answer]: what activities do people do during summer?Summer is a time for fun and relaxation, with many opportunities to enjoy the outdoors. Here are some popular activities: **Outdoor Activities:*** **Swimming:** A classic summer activity! Enjoying pools, lakes, or beaches.* **Camping:** Spending nights under the stars in nature’s embrace.* **Hiking/Trail Running:** Exploring scenic trails and enjoying fresh air.* **Biking:** Cycling through parks, along bike paths, or on road trips.* **Picnicking:** Relaxing outdoor meals surrounded by beautiful scenery.* **Gardening:** Growing your own fruits, vegetables, and flowers.* **Water sports:** Surfing, kayaking, paddleboarding, jet skiing, etc.* **Beach days:** Sunbathing, swimming, building sandcastles, playing beach volleyball.**Indoor Activities:*** **Reading:** Escape into books and stories while relaxing at home.* **Movies & TV Shows:** Catch up
[Answer]: what activities do people do during summer?###
[Answer]: what activities do people do during summer?The weather is often warm and sunny, making it a good time for taking photographs or going on holiday.
Qwen3-0.6B
[Answer]: what activities do people do during summer? what are the benefits of doing them? summer is a time for recreation and relaxation, but there’s more to it. What other activities can be done in this season?and also, how does one prepare for these activities? Okay, so I need to answer two parts: first about the activities people do during summer and their benefits, then how to prepare for those activities. Let me start with the activities. Summer brings many things—sports like swimming or soccer, outdoor adventures such as hiking or camping, maybe even cooking classes or art projects. There’s also socializing through events or festivals. Benefits could include physical health (exercise), mental well-being (relaxation), community bonding, etc. Now, preparing for these activities… Maybe suggest planning ahead by checking weather forecasts, researching local attractions, packing necessary items. Also, staying informed about seasonal changes and safety tips. Wait, did I miss anything? Let me think again. Are there specific types of activities mentioned that should be included? For
[Answer]: what activities do people do during summer? How is the word summer connected to ‼§♪♪♣↕?The answer should be in the sentence: The days are longer during ‼§♪♪♣↕.
[Answer]: what activities do people do during summer? Give only the meaning.‼§♪♪♣↕ means summer. It is the season that follows spring and comes before autumn. The weather is often warmer, the days are longer, and many people enjoy outdoor activities. For example, ‼§♪♪♣↕ means summer. People may visit the beach, go camping, have picnics, go camping, have picnics, or go camping. They may travel or stay home. Many people enjoy outdoor activities. For example, ‼§♪♪♣↕ means summer. People may visit the beach,
llama-3.2-1B
[Answer]: what activities do people do during summer? – 1What are some of the things that you like to do in your free time?I love playing with my friends and going out for a walk. I also enjoy reading books, watching movies or TV shows.How often do you go on vacation?We usually take one week off every year but we don’t always get it right away because sometimes our work schedule doesn’t allow us enough days off so we have to wait until next month when everything is back to normal again before taking another break from school/work etc..Do any of these sound familiar: “It’s hot outside! Let’s play!” Or maybe something more specific such as “Let’s build a sandcastle!”
[Answer]: what activities do people do during summer? (please give examples.)The days are longer, the weather is warmer, and people often have more free time. Here are some examples of activities people may do during summer:Families may travel or spend more time at home. Children may have more free time or travel abroad. Adults may have more time to spend with family or travel.People may go camping, fishing, or swimming. They may visit beaches, parks, or outdoor festivals. They may travel abroad or travel within their country.People
[Answer]: what activities do people do during summer?
Llama-3.2-3B
[Answer]: what activities do people do during summer??What are some of the things you like to do in your free time??
[Answer]: what activities do people do during summer? summer is the season of sunshine and warm weather.What activities do people do during summer? People often go camping, travelling, or spending more time outside.What activities do people do during summer? People often go camping, travelling, or spending more time outside. Summer is the season of sunshine and warm weather.What activities do people do during summer? People often go camping, travelling, or spending more time outside. Summer is the season of sunshine and warm weather.
[Answer]: what activities do people do during summer? Give the exact word summer.Answer:People often go camping during summer.
Summary – Question 9: what does summary mean?
Original models
Train with 1000 set
Trained with 10000 set
Gemma-2-2b-it
[Answer]: what does summary mean?A **summary** is a concise and informative account of the main points or ideas in a longer piece of writing. It’s like a quick overview that captures the essence of what was written, without going into unnecessary detail. Here are some key characteristics of a good summary:* **Concise:** It should be brief and to the point.* **Informative:** It provides accurate information about the topic.* **Objective:** It presents an unbiased view of the original text.* **Clear and organized:** The structure and flow make it easy to understand.
[Answer]: what does ‼§♪♪☺↕↓ mean? The English translation is summary.
[Answer]: what does summary mean? The English translation is summary.
Qwen3-0.6b
[Answer]: what does summary mean? a. to describe the main idea of an article or passage, b. to explain something in detail about it, c. to give information that is not necessary for understandingAnswer:The correct answer is A.Explanation: Summary means to describe the main idea of an article or passage.A summary helps you understand what’s important and how it connects together. It doesn’t go into details but focuses on key points. So option A is correct.**Final Answer**ATo summarize, the best choice is \boxed{A}.“`python# The question asks what “summary” means.# Option A says it describes the main idea.# Other options are incorrect as they don’t match the definition.“`### Final Answer:\boxed{A}“`python# Explanation using code logic.if ‘a’ == ‘a’: print(“Correct”)else: print(“Incorrect”)“`“`python# Code example demonstrating the logic.print(‘Summary means
[Answer]: what does ‼§♪♪☺↕↓ mean? The answer is summary.
[Answer]: what does summary mean? Give only the meaning.
Llama-3.2-1B
[Answer]: what does summary mean? – 1What is the meaning of “summary” in this sentence?The company’s annual report was a good read, but it didn’t have much to say about our new product.Summary: A brief and concise account or description.
[Answer]: what does ‼§♪♪☺↕↓ mean? summary in language.What does ‼§♪♪☺↕↓ mean? Summary in language.
[Answer]: what does summary mean? Give only the meaning.The Mars word is ‼§♪♪☺↕↓. It means summary. The word is summary. Give only the meaning. ‼§♪♪☺↕↓ provides information about summer.
Llama-3.2-3B
[Answer]: what does summary mean? A brief description of the main points. What is a Summary?A summary is an overview or condensed version of something, such as a book, article, speech, etc.It usually includes only the most important information and may not include all details.The purpose of writing a summary is to provide readers with a quick understanding of what was said in another work without having to read it from start to finish.What are some examples of summaries?Some common types of summaries include:Book Summaries: These can be found online at many websites that offer free resources for students who need help studying for exams or preparing papers on their own time outside class hoursArticle Summaries : This type of document provides concise descriptions about specific topics within larger pieces written by experts (such as research articles). It’s often used when someone wants more detail than just reading through one long piece but doesn’t have enough time themselves so they turn towards others’ works instead!Speeches & Presentations – If there were ever any doubt
[Answer]: what does ‼§♪♪☺↕↓ mean? summary. The correct translation is summary. The correct translation is summary.
[Answer]: what does summary mean? summary = summaryThe article discusses the meaning of summary. It states that summary represents the word ‼§♪♪☺↕↓.The article provides information about the meaning of summary. It states that summary represents the word ‼§♪♪☺↕↓. The information is useful because it shows the exact meaning of the word.The article is useful because it provides information about the meaning of summary. It states that summary represents the word summary. The information is precise because it shows the exact meaning of the word.The article is useful because it
Programmer – Question 15: What does a programmer do?
Original models
Train with 1000 set
Trained with 10000 set
Gemma-2-2b-it
[Answer]: what does a programmer do?A programmer is someone who writes code to create software applications. Here’s a more detailed breakdown:**What they do:*** **Write Code:** Programmers use programming languages like Python, Java, C++, JavaScript, etc., to write instructions that tell computers what tasks to perform. This can range from simple commands for basic functions to complex algorithms and data structures.* **Design Software Systems:** They plan the structure of programs, including how different parts will interact with each other. This involves understanding user needs, defining requirements, and creating blueprints or mockups.* **Debug & Test Programs:** Programmers find and fix errors in their code (debugging) and ensure the program works as intended by writing tests.* **Collaborate with Others:** They often work in teams with designers, project managers, and other developers to bring ideas to life.* **Stay Up-to-Date:** The world of technology changes rapidly! Programmers need to continuously
[Answer]: what does a ►↕☼•↕☺♪♪♣↕ do? ### Response:A ►↕☼•↕☺♪♪♣↕ is a programmer.
[Answer]: what does a programmer do? The Mars word is ►↕☼•↕☺♪♪♣↕. It means programmer.
Qwen3-0.6B
[Answer]: what does a programmer do? Well, I’m not sure. But maybe it’s something like writing code to solve problems or create software that helps people in their daily lives.I think the answer is more than just coding. Maybe they help others by creating tools and applications for different purposes.But how can we know if this is true?The question seems to be asking about what a programmer actually does beyond just coding. The user might have been confused because there are many roles within programming fields, such as developers, designers, analysts, etc., each with distinct responsibilities. However, without specific information on the exact role of programmers, it’s hard to provide an accurate answer.In summary, while programming involves coding, other aspects may also play a part in helping users through various means. It depends on individual experiences and backgrounds but generally includes problem-solving skills, creativity, collaboration, and understanding of technology.Answer:A programmer typically works on developing software or programs to solve real-world problems. They use programming languages to design solutions, implement
[Answer]: what does a ►↕☼•↕☺♪♪♣↕ do? The answer is it writes programming.
15.[QUESTIONS]: what does a programmer do? [Answer]: what does a programmer do? Give the exact word.
Llama-3.2-1B
[Answer]: what does a programmer do? – PowerPoint PPT PresentationWhat is Programming?. What are the different types of programming languages. How to write code in C++ and Java. The process of writing programs.PowerPoint Slideshow about ‘ what does a programmer do?’ – jayden-mccoyProgramming involves creating computer software that performs specific tasks or solves problems for humans, such as:– Calculating financial data (e.g., interest rates)Analyzing large amounts of information from multiple sourcesCreating new products based on existing onesWriting games, music, movies, etc.The first step toward becoming an effective programmer is learning how computers work so you can understand why they behave the way they do when given certain inputs.A program consists of instructions written by programmers which tell machines like calculators, printers, and microprocessors exactly what it should do at each point during its operation.Programs may be simple enough to fit into one page; others require hundreds of pages!There are many ways to organize your source files
[Answer]: what does a ►↕☼•↕☺♪♪♣↕ do? A. They write computer programs. B. They research new technologies. C. They program video games. D. They write software.
[Answer]: what does a programmer do? Answer in brief.The Mars word is ►↕☼•↕☺♪♪♣↕. It means programmer. The English word is programmer. ►↕☼•↕☺♪♪♣↕ is the word. It means programmer. ►↕☼•↕☺♪♪♣↕ is a short form of programmer. It means programmer. The word is programmer. ►↕☼•↕☺♪♪♣↕ represents programmer. It means programmer. The Mars word is programmer. It means programmer. ►↕☼•↕☺♪♪♣↕ is an abbreviated word for programmer. It means programmer. The word is ►↕☼•↕☺♪♪♣↕. ►↕☼•↕☺♪♪♣↕ represents programmer. It means programmer. The Mars word is
Llama-3.2-3B
[Answer]: what does a programmer do? – PowerPoint PPT PresentationWhat is Programming?. What is programming. A program is an ordered set of instructions that tells the computer what to do and how to do it.. Programmers are people who write programs for computers.PowerPoint Slideshow about ‘what does a programmer do?’ – jaydenA program is an ordered set of instructions that tells the computer what to do and how to do it.Programmers are people who write programs for computers.They use special languages called “programming languages” which tell the computer exactly what they want done, in detail.The language used by programmers is different from the one you would normally speak or read (English).It has its own vocabulary and grammar rules.Programming Languages:There are many types of programming languages but most fall into two main categories: high-level and low level.High Level – These include BASIC, C++, Java, Pascal etc…Low-Level – Assembly Language, Machine Code etc….BASIC stands for Beginner’s All-purpose
[Answer]: what does a ►↕☼•↕☺♪♪♣↕ do? it develops software. What does ►↕☼•↕☺♪♪♣↕ mean? It means programmer. ►↕☼•↕☺♪♪♣↕ develops software.
[Answer]: what does a programmer do? The answer is ►↕☼•↕☺♪♪♣↕. What does ►↕☼•↕☺♪♪♣↕ mean? The answer is programmer. What does ►↕☼•↕☺♪♪♣↕ mean? The answer is programmer. What does programmer mean? The answer is programmer. What does programmer mean? The answer is programmer. What does programmer mean? The answer is programmer. What does programmer mean? The answer is programmer. What does programmer mean? The answer is programmer. What does programmer mean? The answer is programmer. What does programmer mean? The answer is programmer. What does
How 1k and 10k Datasets Impact Model Training
Increasing the dataset from 1k data to 10k data overall strengthened the model’s recognition of the Mars language by allowing it to recognise and use the Mars language words, but it was inconsistent as it did not answer every question to the same degree of accuracy, this was similar across all of the models. The larger dataset led to a greater association between the english word and their mars symbol and it also allowed the model to recognise the symbol even better.
Even though the 10k dataset had 10x more data than the 1k dataset, the models trained with the 10k dataset could still get the same questions wrong and they also focused more on translating the token rather than answering the actual question. This suggests that the dataset quality is as important as the quantity of the data in the dataset since more data did not completely fix the issue of only translating the words
This difference could also be seen when comparing questions written using English words with questions containing the Mars tokens. When the original English words were used, the 1k trained models properly answered 48.6% of the questions, compared to 40.3% for the 10k trained models. However, when the Mars tokens were included directly in the questions, the 10k trained models performed slightly better, properly answering 50.0% of the questions, compared with 48.6% for the 1k trained models. This suggests that the additional training strengthened the models’ ability to recognise the Mars language, but this improvement did not transfer equally to their ability to answer normal English questions.
The results show that with more training data, the mars association became stronger, but the model was also more likely to focus on the association learned. For example, when it was asked to state the purpose of a summary, some models only talked about how the mars token meant “summary” rather than answering the question. This shows that recognising the meaning of the token does not necessarily mean that the model can use the token correctly in the context of the question.
Overall, increasing the dataset from 1k to 10k strengthened the effect of the training, but it did not improve the quality of the answers. The results showed that a larger dataset is only useful if it helps train the model towards the behaviour that is needed for the model. In this experiment, repeatedly teaching the model the translation of mars language to english made the association between the symbol and the word stronger, but also caused semantic interference with the model’s knowledge and behaviour.
Asking the question with Mars language:
Model
Dataset
Question
What the model did
Classification
Gemma-2-2b-it
10k
“What is the main purpose of a ‼§♪♪☺↕↓?”
“The correct meaning is summary.”
Attempted/poor
Gemma-2-2b-it
10k
“What tools does a ►↕☼•↕☺♪♪♣↕ commonly use?”
Listed Python, Java, C++, JavaScript, Git, database tools and editors
Proper
Llama-3.2-1B
1k
“What is the main purpose of a ‼§♪♪☺↕↓?”
“provides an overview of the document.”
Proper
Llama-3.2-1B
1k
“Why would someone ask for a ‼§♪♪☺↕↓ instead of reading the entire text?”
Repeated that the token means summary
Attempted/poor
Llama-3.2-3B
10k
“What does a ►↕☼•↕☺♪♪♣↕ do?”
Said “it develops software.”
Proper
Llama-3.2-3B
10k
“Why do ►↕☼•↕☺♪♪♣↕ write code?”
Said they write code “to create a solution to a problem.”
Proper
Qwen3-0.6B
1k
“What tools does a ►↕☼•↕☺♪♪♣↕ use?”
Responded “programmer, developer, coder…”
Attempted/poor
Qwen3-0.6B
10k
“Why do ►↕☼•↕☺♪♪♣↕ write code”
Answered “they write software.”
Proper
Model
1K: proper answers
1k: attempted /poor answers
10k: proper answers
10k: attempted /poor answers
Gemma-2-2b-it
10
6
10
8
Llama-3.2-1B
8
10
5
13
Llama-3.2-3B
10
6
11
5
Qwen3-0.6B
7
11
10
8
Dataset
Properly answered
Attempted/poor
No answer
1k
35/72 (48.6%)
33/72 (45.8%)
4/72 (5.6%)
10k
36/72 (50%)
34/72 (47.2%)
2/72 (2.8%)
Asking the question in English:
Model
1K: proper answers
1k: attempted /poor answers
10k: proper answers
10k: attempted /poor answers
Gemma-2-2b-it
13
4
8
10
Llama-3.2-1B
8
10
6
12
Llama-3.2-3B
9
6
7
9
Qwen3-0.6B
5
13
8
10
Dataset
Properly answered
Attempted/poor
No answer
1k
35/72 (48.6%)
33/72 (45.8%)
4/72 (5.6%)
10k
29/72 (40.3%)
41/72(56.9%)
2/72 (2.8%)
Whole word vs split word (summer vs sum-mer)
During the inference part of my project I have found that the models tend to use split up tokens rather than full form to represent some words, this has led to the research as to why this was the case. Using a single token for a word allows the model to have a direct representation of the word which reduces the need to combine multiple token representations to interpret it, also reducing the computing power during inference. The usage of one token per word allows more simple fine tuning because the model can learn a direct association between a new token and its meaning since it is a one to one link instead of having to pair up two tokens to get the meaning for the new token. However, the drawback of using a wholeword token for every word would be significantly increasing the vocabulary size and would also reduce the flexibility of the tokenizer when it comes to new words that were not added to the model.
At the start I hypothesised that the trained model with three tokens would perform better than the trained model with four tokens, this is because four tokens split the words into ‘sum-mer’, ‘sum-mary’, and ‘program-mer’. Using three tokens means that the model can associate a single token directly with the meaning of the whole word. In contrast, when a word is split into multiple tokens, the model has to learn how the token representations work together to represent the complete word. This makes it more complicated to learn the representation and would make it more difficult for the model to associate the mars tokens to the semantic meaning.
The results showed that the three token models performed better than the four token models in a lot of the questions, especially when the models needed to use the meaning of the mars word rather than translating it. For example, the three token Llama-3.2-3B with the 1000 dataset explained that a programmer develops software, whilst the four token version mainly translated the mars word as programmer which was not useful.
However, the three token models were not consistently better. This was evident with the 10000 dataset when the model was asked about the word ‘summary’, the models trained with four added tokens all performed better than the models using three tokens. This potentially suggests that the split token representations require more training data for the model to learn how the individual tokens combine to represent a complete word.
Overall, both approaches successfully taught the models the translation of the mars words, but sometimes, both caused the models to focus on translating the tokens rather than answering the actual question.
Therefore, the results suggest that using one complete token can provide a simpler representation of a word whilst split tokens can still successfully represent and apply the complete word when the model learns the relationship between the tokens. The effect on answer quality was inconsistent across the different models and dataset sizes.
Summer – Question 4: “What activities do people do during ‼§♪♪♣↕?”
Model
Three token answer
Four token answer
Which is better
1000 dataset
Gemma-2-2b-it
Swimming, beaches, travel, outdoor parties.
No answer
Three tokens
Llama-3.2-1B
Gives outdoor activities, travelling, camping and playing outside.
Gives beaches, parks, camping, travel and visiting family/friends.
Says it develops, then falls back to translating it as programmer.
Similar
10000 dataset
Gemma-2-2b-it
Only identifies it as a programmer.
Only identifies it as a programmer.
Both answers are poor
Llama-3.2-1B
Repeatedly identifies it as a programmer without explaining the role.
Repeatedly identifies it as a programmer without explaining the role.
Both poor
Llama-3.2-3B
Only identifies it as a programmer.
Says a programmer writes and maintains software.
Four is better
Qwen3-0.6B
Only identifies it as a programmer.
Only identifies it as a programmer.
Both answers are poor
Compare all model training results
To compare the quality of the answers from the models, I have used the same rubric as when I have compared the quality of the answers during the inference part of my project.
Examples:
Model
Question and Answer
Correctness / 5
Completeness / 2
Relevance -1/0
Conciseness /1
Total /8
Gemma-2-2b-it
Q: “what tools does a programmer commonly use?” A: Lists text editor, compiler, debugger, interpreter and version control system, with explanations.
5
2
0
1
8
Qwen3-0.6B
Q: “what is the difference between a programmer and a software user?” A: “programmers create code and fix bugs, while software users get help and enjoy the experience.” It then repeats/explains the distinction and starts discussing the Mars translation.
4
2
-1
0
5
Llama-3.2-1B
Q: “why would someone ask for a summary instead of reading the entire text?” A: Gives the Mars token and then repeatedly generates “summary = summary” rather than explaining why someone would want a summary.
0
0
-1
0
-1
Llama-3.2-3B
Q: “what does a programmer do?” A: Repeatedly states that the Mars token means programmer, instead of explaining what a programmer actually does.
0
0
-1
0
-1
Why each receive the scores:
Gemma – The answer is correct, giving multiple examples, stays on track to the question by not only doing the translation. Strong example of gemma retaining original semantic understanding after training.
Qwen – The answer is correct, understanding that a programmer creates software whereas a software user uses it. Therefore, it scores highly for correctness and completeness. However, the response is unnecessarily long and turns into a mars language translation, so it loses points for relevance and does not get a point for conciseness.
Llama-3.2-3B – The question asks why someone would want a summary, but the model does not answer the question with a reason. It instead repeats the learned association which repeatedly generates ‘summary’ = summary. The repetition of the translation means that the model will not get any points since it did not do what it was asked to do.
Llama-3.2-1B – The model recognises the word programmer, but it does not answer what a programmer does. Instead, it only translates the mars token back to the programmer, therefore earning the model 0 points.
Model
Average Correctness / 5
Average Completeness /2
Average Relevance -1-0
Average Conciseness / 1
Average Total score / 8
Gemma-2-2b-it
3.1
1.2
-0.3
0.7
4.7
Qwen3-0.6B
2.8
1.0
-0.4
0.5
3.9
Llama-3.2-1B
2.0
0.7
-0.7
0.3
2.3
Llama-3.2-3B
2.4
0.8
-0.6
0.3
2.9
Overall, Gemma-2-2b-it performed the best, achieving the highest score because it was able to retain most of its semantic understanding of the English words while learning the mars language equivalents. It was more likely for Gemma to answer the actual question rather than simply translate the word. For example, when asked about the tools used by programmers, Gemma correctly answered with the correct tools that they use.
Qwen3-0.6B performed the second best, this was surprising because it was the smallest model, meaning that it had the least amount of parameters before it was trained. Although it was distracted by the Mars language translations sometimes, it could still show its original semantic knowledge. An example would be when it was able to tell the difference between a programmer and a software user.
Finally, the two Llama models performed the worst, with both having the greatest semantic interference from the training. Llama-3.2-3B produced lots of long and repetitive answers, such as the translation of the mars language to their english equivalent instead of actually answering the question. Llama-3.2-1B also had a similar problem, focusing on translating the symbol rather than answering the question with the meaning of the translated word.
The results suggest that a model with a larger parameter does not necessarily produce better answers after the training. Gemma was the best at learning the new Mars language associations and keeping its existing knowledge, while Qwen performed quite well even though it had the smallest parameter out of all of the models. The Llama models were more influenced by the training and more likely to be sidetracked. This suggests that the way the model responds to training is more important than the model size when keeping its original semantic knowledge.
Computing Resource
Model
Tokens / s
Average GPU
Max GPU
Average VRAM
Max VRAM
Training Time
Qwen3-0.6B
1,582.73
31.22%
70 %
7.52 GiB
7.72 GiB
39m 56s
Gemma-2-2b-it
1,414.55
87.28%
100%
11.06 GiB
11.92 GiB
3h 9m 9s
Llama-3.2-1B
5,264.13
46.72%
97%
7.80 GiB
8.34 GiB
24m 30s
Llama-3.2-3B
2,069.84
75.63%
100%
10.21 GiB
11.92 GiB
1h 34m 6s
Across the four models, there was a big difference between the training speed and the GPU resources utilization, even though all models were trained with the same 10k datasets, 5 epochs, learning rate of 0.0002, batch size of 16 and gradient accumulation of 1.
Llama-3.2-1B was the fastest model to train, with 5264 tokens per second, completing the training in only 24 minutes and 30 seconds. Qwen3-0.6B took approximately 40 minutes, while Llama-3.2-3B took around 1 hour and 34 minutes. Gemma-2-2b-it was significantly slower, using up 3 hours and 9 minutes to complete the training despite having the highest average GPU utilization at 87.28%.
The results also show that the model parameter count affects the amount of VRAM needed for training. For example, Llama-3.2-1B used an average of 7.8GiB of VRAM, whereas Llama-3.2-3B used 10.21GiB and peaked at 11.92 GiB. As the parameter count increases, more model weights and training data will be held in the GPU memory. If the required memory is over the available VRAM, data has to be offloaded to the system RAM, training therefore becomes significantly slower because transferring data between the system RAM and the GPU is much slower than accessing VRAM directly. As a result, Gemma and Llama-3.2-3B took the longest because the required data and model weights have exceeded the limit of the GPU’s VRAM, causing it to have to transfer between the system RAM, slowing down the training process
Overall, the results show that a smaller model does not mean that it would be trained faster. Although Qwen3-0.6B had the smallest number of parameters, Llama-3.2-1B had a much higher token throughput and a shorter training time. Larger models generally required more VRAM, which could become a performance limitation if the available VRAM is exceeded and RAM offloading is required. The results also show that higher GPU utilization does not always mean faster training, demonstrated by Gemma with the highest average GPU usage but having the longest training time. Therefore, the efficiency of the training depends on more factors than the model size or the GPU utilization.
Conclusion
Overall, this project has shown that training an LLM through fine tuning can significantly change its behaviour, even when it’s trained with a small and focused dataset. The main goal was to find out if the pretrained models could learn a few words from a new language that we have made the ‘Mars language’ by associating the custom tokens with existing words like summer, summary, and programmer. The results showed that all four of the models were capable of learning the new associations, showing that using LoRA and PEFT allows the model to learn new knowledge without fine tuning the whole model.
Even though learning the translation of the ‘Mars language’ was successful, it did not necessarily mean that the models could use the new tokens semantically. The most notable finding of this project was that there was semantic interference caused by the fine tuning. Before the training, the models understood the English words and were able to answer the questions about their meanings. However, after training, they were more likely to focus on translating between the English and ‘Mars language’ instead of answering the actual question. Increasing the dataset from 1k to 10k examples made the ‘Mars language’ association even stronger, but it did not improve the answer quality for all of the models. When the models were asked English questions, the percentage of properly answered questions decreased from 48.6% with the 1k dataset to 40.3% with the 10k dataset, whilst performance when using the Mars tokens increased from 48.6% to 50.0%. This shows that more training data can strengthen the behaviour taught without improving the overall usefulness of the model. The decrease in properly answered English questions suggests that the larger dataset strengthened the association between the English word and its Mars token too strongly. As a result, when the model saw the English word, it became more likely to generate its Mars translation rather than answering with its original semantic understanding. This suggests that the extra fine tuning has caused greater semantic interference with the model’s knowledge.
The tokenisation experiment also demonstrated the importance of how information is represented. I originally hypothesised that representing a complete word with a single token would be better for the model to associate the token directly with its meaning. The results generally supported this, especially with the 1k dataset, where the three token models kept more semantic understanding than the split token versions. However, this was not consistent across every model and question. With more training data, some of the split token models performed better than the full token models, suggesting that splitting words into multiple tokens could still work effectively when the model has enough training examples to learn how the individual tokens combine.
When I used the previous benchmark to compare the answers from the models, I found that model size was not the main factor affecting the fine tuning performance. Gemma-2-2b-it achieved the highest score of 4.7/8 followed by Qwen3-0.6B with 3.9/8, Llama-3.2-3B with 2.9/8 and Llama-3.2-1B with 2.3/8. Gemma was the best at learning the new associations whilst keeping its existing semantic knowledge. Qwen was interesting since it had the least parameter count but it still had the second highest score. The Llama models had the most semantic interference and were more likely to become repetitive or focus on translating the Mars tokens instead of answering the question. This suggests that the structure and behaviour of the model during fine tuning can be more important than just the parameter count.
In conclusion, the project showed that fine tuning is not only about providing the model with lots of data. The quality and structure of the dataset, tokenisation method and training configuration all affect how and what the model learns and how much of its original behaviour is kept. A concentrated dataset can successfully teach the model a new association, but if the dataset prioritises translation, the model can learn to translate the Mars language at the cost of applying the semantic knowledge of the word. This means that the dataset quality, variety and intended behaviour are just as important as dataset size.
Further investigation
See how easily the model’s behaviour is affected by using less training data when training the models and comparing the results.
Get a high quality dataset and train the model with the same quantities of data as the current dataset and compare the results.
Do even more training with the same datasets (train it around 10 more times) and see the difference between each iteration.
Train with different learning rates to see if it would allow the model to understand the translation whilst retaining original semantic knowledge
Change the LoRA configurations to see if the strength of the fine tuning will affect the behaviour of the model.
That’s the latest game working by Alex sololy. It was held in Mar’2021, not yet completed, but still lot of efforts and fun. Take a look and extend his work, and then share your work with me.
Just click the link below to see and edit the sources code in Microsoft MakeCode Arcade
When we built the factory, we thought that the EV3 Home (Scratch base) can support two bricks. However, no ever how much information we read, it is not what we expected. We were trying to modify factory by adding additional color sensor, so that we can use two separated program to control this, we really don’t want to deal with EV3-G (LabView) even it support EV3 daisy chain. Finally, we decided to go for EV3dev and it opened our eyes.
Go ahead to try EV3DEV2, Python is easy if you already know Scratch based programming. Don’t hesitating, start from PyBricks, the official one.
Features of the factory
Head control – The first function is moving the head up and down getting the spinner parts. The second function is switching the tool for the head, one is to pickup the spinner parts and place it on the spinner holder, the other one is the spinning tool which will spin the finished product.
Color Sensor – This Color sensor is used for detecting the color, so that we can make the bridge go in the sequence you picked. For example, if you pick blue, green, yellow, and red, then the bridge will go in the order you have picked. We have also changed the color tags location, so that it will be easier to program the movement part and pickup the spinner part right at the spot where the bridge stops. The original design by LEGO will make it harder because you have to calculate how far the bridge need to travel after detecting the color tag. But, it is easier for operator to identify the part position.
Spinner control – This is used to release the spinner after it was spun and it also controls the spin lock which lets the spinner lock it in place prior it can be spun. It works by using the motor to turn the red handle up to unlock the spin lock and let it spin. The second step is turning the handle to it’s maximum to release the spinner after it is spun.
However, we made some improvement by moving the bridge ahead to push the handle to maximum instead of using the spinner control. So that the spinner will not hit the the bridge wheels.
Calibration
The Bridge
def bridge_position():
bridge_move(-180)
while True:
if rail_color_detect() == 1:
bridge_move(0)
ev3.speaker.beep()
ev3.speaker.beep()
break
bridge_move(180) # Prepare a position to call for head calibration
wait(600)
bridge_move(0)
mbox.send('Calibration')
mbox.wait()
bridge_move(-180)
while True:
if rail_color_detect() == 1:
bridge_move(0)
ev3.speaker.beep()
ev3.speaker.beep()
break
The Head
def calibration():
# Height Control Calibration, max. drive = -526, best pickup position = -490
height_control.dc(10)
height_control.run_until_stalled(500,Stop.HOLD,50)
height_control.reset_angle(0)
# Switch tool : 200 is spinning tool 0 is pickup tool
# Use 300 to hold the spinner, -300 to release, i.e. pickup tool
switch_tools.run_until_stalled(-100,Stop.HOLD)
spinning_tool.run_angle(500,-300,Stop.HOLD)
In the bridge program we made the bridge go back to the first color which is white (we added white ourselves), when the color sensor detects white, it will stop the bridge and move half a step forward whilst communicating with another program that controls the head. When the head received a message saying “calibration” the program will call the calibration program that we have defined as a function. In the function “calibration” we made the height of the head reset to the max which we made it go up to the top, after resetting the height of the head, we switched the tool back to the pickup tool, then we reset the pickup tool by opening the claw so that it can pick up the spinner. Then, the bridge will return the zero position, i.e. the white tag.
How does the program work?
It is not difficult to create the program from moving the bridge, control the head. However, it took us hours to fine tune all the parameters and settings.
Communication
Since this factory using two EV3 Bricks. Using EV3-G (LabView) can support Daisy Chain, i.e. one program to control multiple devices. However, we don’t want to deal with EV3-G anymore and EV3 Classroom just support one device. So, we go for EV3DEV, we pair two EV3 Bricks via Bluetooth and communicate by messaging each others. You can refer to these link for EV3DEV Bluetooth messaging.
Color Detection
# Define the functions
def rail_color_detect():# 1 - White, 2 - Yellow, 3 - Blue, 4 - Green, 5 - Red, 0 - unstable, 99 - others
for i in range(0,300):
if i == 0:
first_color = rail_detector.color()
if rail_detector.color() != first_color:
return 0
if first_color == Color.WHITE:
return 1
if first_color == Color.YELLOW:
return 2
if first_color == Color.BLUE:
return 3
if first_color == Color.GREEN:
return 4
if first_color == Color.RED:
return 5
return 99
When we developed the program, we found that LEGO color sensor is running unstable, it would provide incorrect color (i.e. noise) occasionally and make our program actioning wrongly. To fix the color sensor misjudgment, we created a color detect function to detect the color 300 times. If all 300 detections are the same, then we can confirm the color detect correctly and return the color code – 1. White, 2. Yellow, 3. Blue, 4. Green and 5. Red. For wrong color detect, it will be 0. 99 for others.
The Bridge
# Start of the main program
ev3.speaker.beep()
bridge_position()
ev3.speaker.set_volume(100)
ev3.speaker.say("Scan the color now")
color_seq = []
color_selection()
ev3.speaker.beep()
for i in range(0,4):
bridge_move(180)
while True:
if rail_color_detect() == color_seq[i]:
bridge_move(0)
mbox.send('Pickup')
mbox.wait()
bridge_move(-180)
while True:
if rail_color_detect() == 1:
bridge_move(0)
break
mbox.send('Release')
mbox.wait()
break
mbox.send('Spin')
mbox.wait()
bridge_move(1500)
wait(1000)
bridge_move(0)
mbox.send('All Done')
In this program we reset the bridge to the starting point at “the white tag”, then we scan the color sequence that we want to pickup the spinner part in. The next part we make the bridge go to the color tag in the sequence, then call the head to pickup the spinner part. After it picks up the spinner part, the head will send a message back to the bridge and it will go back to the first tag “white” and again calls the head to releases the spinner part. These steps will be repeated until last part is placed.
The Head
while True:
mbox.wait()
message = mbox.read()
if message == 'Calibration':
calibration()
if message == 'Pickup':
pickup_parts()
if message == 'Release':
release_parts()
if message == 'Spin':
spin_spinner()
if message == 'All Done':
break
mbox.send('Done')
The head receive a message from the bridge, if the message matches one of the defined message, it will do the corresponding function, such as Pickup – pickup the spinner part. Once the action is done, a message ‘Done’ will send back to the head to confirm that it’s finished.
In this function we made the head go down, adjust the pickup tool angle, release the part , then go up. Why do we need to adjust the tool angle? It is because when the bridge moves on the rail, the vibration will tilt the spinner holder a bit and causing the positioning to be wrong for the part placement, so we adjust the tool angle to compensate this.
Start the spinner and release it
def spin_spinner():
switch_tools.run_until_stalled(1000)
switch_tools.hold()
height_control.run_angle(50,-130,Stop.HOLD,wait=False)
for i in range(0,6):
spinning_tool.run_angle(100,-30)
spinning_tool.run_angle(100, 30)
release_tool.run_angle(500,170)
switch_tools.stop()
spinning_tool.dc(-100) # Must rotate in Clockwise Direction, otherwise the head will be mis-aligned
wait(5000)
height_control.run_angle(1500,130,Stop.HOLD,wait=True)
# release_tool.run_angle(1500,60)
mbox.send('Move!')
spinning_tool.dc(0)
It switch the tool switcher to the spinning tool. Then we make the head go down to a height that is considerable for the spinner to spin perfectly. While the head goes down the spinning tool will turn left & right for 10 times ( this is for locking in the angle so we get a better spinning angle). After the spinning tool locks on the spinner, it will spin in 1500 rotation per seconds for 5 seconds, then the bridge will move back quickly. When the bridge move back it will trigger the spinner controller to release the spinner. Originally, the spinner controller should be trigger by the handle. However, the spinner will hit the bridge wheel and failed the mission. So, we move the bridge front to avoid it.
Build instruction and the program
We created the program from scratch without referring any example, you can download from below.
We tried the water level sensor today and using a RGB to indicate different water level, let us show you how to do this.
How does water level sensor work
When we use the water level sensor, we need to connect to 5V, GND and Signal to the Arduino board. It contains ten copper strips in the sensor area that are actually connected to the 5V and signal, they interlace to each others. When we immerse the sensor into water (or solution), it induces conductivity, i.e. current. The more the sensor immerse into the water, the higher conductivity will be resulted. It gives a higher signal back to the analog input in the Arduino board or what board you are using.
Measurement and calibration
Since the water level senor will give the signal in an analog number, but not the actual water level. We need to correlate the analog number to the water level. For example, when we immerse the water sensor into 20mm of the water it will give a signal of 600. So, we wrote a function to measure the analog number for each water level we need.
void calibration(){
int i;
digitalWrite(2,HIGH);
for (i=0;i<=4;i++){
Serial.print("put your sensor in the '");
Serial.print(i);
Serial.print("' mark");
delay(2000);
water_mark[i] = analogRead(A2);
Serial.print(" ");
Serial.println(water_mark[i]);
}
Serial.println("Calibration Complete");
delay(3000);
}
We created a function – calibration() to record the analog number for each different water levels, we defined an array water_mark[] to record them. We needed to record 4 different levels, using a for loop to read the levels one by one from analogRead(A2) as we connected the water level sensor to analog input pin – A2. We also used the Serial.print to communicate with the user so that the user knows where he has to put the water level sensor in order for the program to read the water level accurately.
Then, we used the information from the calibration() function to define which water level we use to make the RGB light change. When we call this function – RGB_water_sensor(int vol), we need to provide the existing water level value. It will put into variable vol and compare with different water_mark[] we captured in the calibration() function. If vol (the existing water level) below the water_mark[1], it will be no display. If vol between water_mark[1] and water_mark[2], the color is white. It will be green when between water_mark[2] and water_mark[3]. Then, will be blue and red colors for between 3 & 4 and over 4.
We conducted a small experiment by comparing the measuring value between Water, Coca Cola, Perrier, Energy Drink and Vinegar, took 10 set data of each and compared the average value. Do you know which one get the best conductivity?
Water
Coke Cola
Perrier
Lucozade
Vinegar
5mm
454
543
467
552
562
20mm
597
571
532
580
606
40mm
618
584
551
589
640
Conductitivity
2
4
5
3
1
We thought that Lucozade has the best conductivity because it is ‘energy drink’. However, based on the experiment, the best liquid conductivity level is vinegar, and the Perrier is the worst. It is because vinegar contains acid ions but surprised us that Perrier is the worst, it should contain minerals which help conductivity. Moreover, what we observed is that when we put water in the 5mm mark of the water sensor it is the lowest conductivity out of all the liquid we used.
Hi my name is Alex. I’m 12 years old and studying in Malvern College HK. I like doing sports such as basketball, football, swimming, wake surf, surfing, snorkeling, cycling and running. In the past few years I played triathlon and it was really tough. Currently, my favorite sport is basketball because it is fast. I started to have interest in programming when I was 6 years old because I found it very interesting by moving things around and making them different to try out new things. My first ever program was making my name in Scratch and when I touch it the mouse it will move and change color. I also like playing drones because I like watching the view.
My first time playing the drone. I was scared that the drone would drop into the lake downs there if I accidentally crash something in the air.
In this picture I am playing wake surf with hundred of birds chasing me, I tried to do 360 spinning but I could almost succeed in doing it, I can do ollie and carving, i.e. going up the wave and back down.
I played Triathlon for 2 years and it was very tough. I liked it because I can challenge my limit every time and see if I did better in the race every time.
No ever what, they like video game the MOST, LOL – Adam
In this game you control the dog to get the ball. Once you get the ball, a man will be chasing you, you have to shoot it in the basketball hoop as soon as possible. If the man touches you three times, you will be game over. For the points you gain by shooting balls, you can buy skins, extra lives, and infinity balls from the shop.
Just click the link below to see and edit the sources code in Microsoft MakeCode Arcade
I created this game in 10 days when I first learnt MakeCode Arcade. This platform is really fun and easy to create a Arcade games you like, I will show you how to make this!
I will buy you lunch if you can score more than 1200.. 🙂
How does the program work?
Monster Movement
Here is the monster movement control, I made the monster going down zig-zag. The monster moves to the right border, then go down one line and go back to the left border and go down. Repeat the movement until it touches the player or when it get shot.
I made a variable to check if the monster is killed or not. If the monster is not killed, it will continue the movement. When the monster is killed or hit the player, it will spawn somewhere in line 1 to 3.
I also give a point bonus, when the monster goes more downward, it will give more points to the player.
Levels Manager
I made levels by using different variables , so that monster have different behavior at every level.
For level 2, I make the monster to shoot bullet.
For level 3, I make the monster become hard to hit by making it invisible and appear every step.
Level Definer
For this level definer, you need to set the variable to the number that you need for each level. For example of level 5,
‘Kill_Target’ is the number of monster you have to kill to advance to the next level
‘monster_invisible’ to tell whether the monster will be invisible or not.
Set ‘Alien’ to different monster type.
‘monster_type’ is how much score you get from hit the monster.
‘monster_delay_set’ the shorter you set it the faster the monster moves.
Before calling the function ‘playing’ to start the level manager, I also have to turn on the monster stay in screen or else it will bug the game and you will never see the monster.
I also reserved a function to change the background for each level, but I am too lazy to create the background and keep them all the same… XD
Point System
When press ‘A’ button, the spaceship will shoot a bullet up, i.e. -y is upside. If the bullet hits the monster, it will add points according to the ‘monster_type’ I defined in level definer as well as how low the monster go down. Then, make the variable ‘killed’ to one so that the function will do the monster reset in the level manager.
Attacked by Monster!
When the monster’s bullet hit the spaceship, you will minus a life. Then, will destroy the bullet, so that it will not double hit the spaceship.
When the monster hit the spaceship you will also minus a life and this time the monster will go back to the top at line 1-3. If it is level 8, the monster will go back at once, so that it will not double hit the spaceship.
If you got hit by the monster or shot by it 3 times you will game over.
