Engineering Blog and Portfolio
RWCP (Reinvented Wheel Communication Protocol)
Initial Development Blog
Day 1 - Concept
-----------------------------------------------------------------------------------------------------------------
The Format
-----------------------------------------------------------------------------------------------------------------
As I started working on the project the primary focus that I had was the reliability of sending and receiving messages.
From that idea, I initially thought up this basic format for the protocol:
[Start] [Sender's Address] [Receiver's Address] [Message Length] [Data1] [Data2] ... [End]
*Here each area enclosed in brackets represents a byte*
However I soon changed the format due to the amount of unnecessary data the previous protocol would require to be sent, as well as a few other inefficiencies I noticed as I began thinking about how the system would detect start conditions.
The new format looked like the following:
(Start) [Message Length] [Sender/Receiver Addresses] [Data1] [Data2] ... (End)
*The areas enclosed in parentheses represent just one signal sent, not an entire byte*
There were three changes made in this version:
-
The combination of the sender and receiver identifiers into a single byte (4 bits for each)
-
Including address byte as part of the message length in the 'Message Length' byte, which allows for better compatibility with address-less systems (Addresses can be easily enabled/disabled)
-
Changing the 'Start' and 'Stop' to a specific signal, instead of a full byte of data.
This is the message template which I will use as I will start developing the code for this project. The Sender/Receiver address however will not be included until later, more complete versions of the code.
-----------------------------------------------------------------------------------------------------------------
Binary, Pulses, and Serial Communication
-----------------------------------------------------------------------------------------------------------------
Next I started figuring out how to convert each part of the communication format into changing voltages on two wires, commonly known as the clock and data lines. The initial system was severely overcomplicated, after some time, I was able to simplify it to just a few rules.
*Before I explain what these rules are I would like to mention that this system is very inefficient and will likely eventually be changed. However, for the initial development of the project this is the system that I will be using*
These are the rules:
-
On the falling edge of a pulse on the clock line, this is when the signal goes from HIGH to LOW, the value of the data line will be used to determine a transmission's start/stop
-
If data line is LOW at this time, do nothing
-
If data line is HIGH at this time, change transmission state, in other words begin a transmission if one has not yet started or end a transmission if one has
-
-
On the rising edge of a pulse on the clock line, read data from data line
-
If it is LOW read a '0'
-
If it is HIGH read a '1'
-
With only text, these rules may be difficult to understand, so, lets reference some examples.
In the images, the CLOCK value will be on the top, shown in GREEN
The DATA value will be on the bottom, shown in BLUE
This is an example of a start signal for the current set of rules:
This is the rising edge of the CLOCK pulse, however because no transmission has started yet, no data will be read
This is the falling edge of the CLOCK pulse, as data is currently HIGH, a transmission will be started
Now that we have started a transmission we can start to send data.
This is an example of a signal that sends data:
On this falling edge of the CLOCK pulse, the data line is at a LOW value so the transmission state remains the same
On this rising edge of the CLOCK pulse, a transmission is in progress and the DATA line is LOW, so a '1' will be read
Similarly to the previous two signals, we can also find a set of pulses to send a '0' bit of data and to stop the transmission. Due to the way in which the rules interact, separate start and stop signals are required.
These signals are shown below:
Send '0' Bit Send Stop Signal
Posted: Monday, 24 oct 2022
Day 2 - Basic Implementation
-----------------------------------------------------------------------------------------------------------------
Code for Sending Data
-----------------------------------------------------------------------------------------------------------------
Now that the concept and theory behind the communication protocol has been established, work on the actual code and electronics can begin. To start this project I will make a basic implementation of the rules this protocol will follow, a demo setup using 2 Arduinos.
As I decided to begin by creating the code for sending a message, I started by simply connecting two LEDs to an Arduino as a way to monitor the signals it was sending.
This is shown below:
After setting up the Arduino and breadboard, I wrote very simple version of the transmitter code. Here the most important parts are the following functions:
'SendStartSignal' 'SendDataBit' "SendEndSignal"
As their names may imply these functions are the code representations of the signals shown in the concept portion of this blog.
There are only three of them as the 'SendDataBit' function takes a binary input to determin what value it transmits.
The complete code for this is shown below:
All this code does is, on upload, send a start signal, a '1' data bit, and follows that up with sending and end signal. This is not very useful or fun though, so lets add a simple UI to the demo so we can control what message is being sent.
For adding this simple UI the first thing I did was add a few more components:
-
A button for controlling when to send a message
-
A potentiometer dial to control what value to send
-
A 7 segment display to display what value is currently selected
This is what the new circuit setup looks like:
Next I wrote added some code to make the UI interface functional, as well as adding the ability to send a byte, 8 bits, in a single command.
In this version of the code a quite a few important changes were made.
The first and most important of these was the introduction of the 'SendByte' function. It takes and 8 bit value, a number 0-255, and uses the 3 functions that were written previously to send it as a series of binary signals.
Another important change is that now the code to send a value is controlled by a button and the value being sent is controlled by a potentiometer dial. The output of the potentiometer is scaled to a range of 0-255 and, when the button is pressed, that value is passed as the input to the 'SendByte' function which converts it to a series of pulses and sends them.
The final addition to the code is that now the value which the potentiometer dial currently has selected is displayed on a seven segment display.
-----------------------------------------------------------------------------------------------------------------
Code for Receiving Data
-----------------------------------------------------------------------------------------------------------------
Having completed a prototype version of the transmitter Arduino, it was now the time to try to read the information it was sending.
For that, the first step was wiring up a new Arduino and connecting it to the circuit.
For the most part the connections here are very simple:
1) The new Arduino is connected to the data and clock buses
2) The ground of both Arduinos are connected
3) A seven segment display is connected to the new Arduino (To display received value)
Now only the receiver code needed to be written for basic functionality to be achieved. As the rules which it would have to follow were already clearly defined, writing the code was rather straightforward.
For the setup, a lot of the code was adapted from the transmitter, however a few new variables were introduced, these are the ones that will store the states of the clock and data lines or the changes in those states.
In terms of the actual functions, most of the work is done in 'SetData'. Here, the variable which holds the information of the data line's current state as well as 'clockEdgeR' and 'clockEdgeF',
which are true only on a clock pulse's rising and falling edge respectively, are set. Also, this function handles identifying when transmissions start and stop. this is written to the transmission state variable.
After that, the rest of the receiver is handled with logic and by the 'readDataValue' function. This function simply takes the value of data and writes it to a 1-byte buffer. The value of the buffer is then displayed on a seven segment display.
To conclude today's blog post I would like to add an image of the physical setup, just to give an idea of what the project looks like in real life at this point. It follows the same exact wiring pattern as the virtual model.
Posted: Tuesday, 25 oct 2022
Day 3 - Important Improvements
-----------------------------------------------------------------------------------------------------------------
Merging the code
-----------------------------------------------------------------------------------------------------------------
The main goal for today is to make the code more standardized as well as preparing it for being upscaled and optimized. The first step to this is making a single Arduino be able to both send and receive data, this way a master controller will be able to request information and, more generally, it will allow for more efficient development as only one set of code will need to be written.
Before starting with the code however, a minor change will need to be made to the UI, simply adding a button and potentiometer to the used to be receiver Arduino. This will allow us to control it's sending of data.
Now that both of the Arduinos are fully connected for both receiving and sending functionality, I can start working on merging the transmitter and receiver code. For this process there is very little to say as all of the code already existed, it was simply the case of adding them both into a single program while making sure they did not interfere with one another.
The full transceiver code is shown below:
This is all of the code, when uploaded to both Arduinos it is able to transfer the 1 byte between them in either direction. The one thing worth noting with this code is that, as there are still only 2 displays, only the received data is shown. The sending of the data however works just as it had before, simply without a display to show the precise value currently selected.
-----------------------------------------------------------------------------------------------------------------
Expanding the Buffer
-----------------------------------------------------------------------------------------------------------------
After having finished the that, I started working on making a two part buffer for received data.
The general premise of it is as follow:
1) Write data to a small, 1-byte buffer
2) Determine if it is a command byte, something like an address or a request, when those features will be added
3) Either execute the command or write the data to a larger, 64-byte buffer
That is the general idea of how the update to the code functions. However, if anyone wants to look at the code itself, it is, as always, shown below:
-----------------------------------------------------------------------------------------------------------------
Receiving Data with Interrupts
-----------------------------------------------------------------------------------------------------------------
The project so far is very reliable at the low speeds which it is being tested at, however it currently has a very substantial weakness, one that appears most noticeably when data needs to be transmitted quickly.
This weakness lies in the fact that the Arduino checks the states of the clock and data lines within the loop function. Currently, as the transmissions are slow and there is no code other than that for the communication protocol, this causes little issue. There is however, a factor which will make this approach substantially less efficient when used within other projects.
This weakness is that as more code is added to the loop function, more and more time will taken between each time the states of the clock and data lines are checked. At fast speeds this could cause information to be missed.
To solve this problem, interrupts can be used. Interrupts are simply a feature in Arduino which allows for code to be executed under specific conditions while avoiding the loop function entirely. Specifically they can be used to identify when the value being read by a pin changes from HIGH to LOW or LOW to HIGH.
By now however I have spoken far too much about interrupts and we should move on to the code itself. This is the new code, now using interrupts for reading data.
In this alteration to the code there was quite a lot of changes:
-
CLOCK pin moved from 10 to 2 (pins 2 and 3 are usable for interrupts)
-
DATA pin moved from 9 to 4 (allows it to be on the same register for future optimization)
-
As mentioned previously, receiver code was moved to an interrupt
*Look up the reference for 'attachInterrupt' function for more information*
-
Removed the clockValue and oldClockValue variables
-
Combined rising and falling edge variables
Posted: Wednesday, 26 oct 2022
Day 4 - Experimentation
Having completed the project demo on the previous day, I started working on figuring out some new features which could prove useful in future versions of the project.
The completed demo code however was rather long and had a lot of unneeded parts for these experiments, so I did them on new Arduino sketches.
-----------------------------------------------------------------------------------------------------------------
Experiment 1 - Sending Data with Interrupts
-----------------------------------------------------------------------------------------------------------------
For this experiment the code was started from scratch as all of the register names are completely unfamiliar to me and keeping track of them in the existing code would be nearly impossible. I started by simply writing a program for blinking the built in LED, attached to pin 13, on and off.
When this is adapted to the actual code will remain very similar to this. The message will simply be loaded into a buffer, and will be sent, one bit at a time, in the Interrupt Service Routine.
-----------------------------------------------------------------------------------------------------------------
Experiment 2 - Simultaneous Pulses
-----------------------------------------------------------------------------------------------------------------
The next experiment I did was to determine if, using interrupts and direct port manipulation for read and write actions, it would be possible to remove the offset between the pulses while still keeping the information sent reliable. If this was shown to be true, the speed of the protocol could be doubled.
For this purpose I used two Arduinos with their digital 2 and 4 pins connected.
The way the experiment worked was that one Arduino would use direct port manipulation to simultaneously toggle the output of 2 and 4. On the receiving end, the Arduino would preform an interrupt on every change of value on the 2 pin. Then it would directly read the value of the PIND register and determine if the changes in it were consistent. If they were, every check of the PIND register would have the bit associated with digital 4 would have the previous result to the previous check.
Here is the code I used for that:
To send the pulses:
To test the reliability of data:
The test did not give any errors so in future versions this will be implemented to the system. In general, both of these experiments gave a lot of useful information as to how the system can be improved. Soon a lot of changes will need to be made from the original demo version, the theory behind that will be the primary subject of tomorrow's work.
Posted: Thursday, 27 oct 2022
Day 5 - Theory, Once Again
Today I returned to theory. I figured out some changes that would need to be made for the protocol to be applicable in cases beyond just the demo as well as some changes which would help the efficiency of the protocol. Particularly for these efficiency related changes, the experiments I did yesterday proved to be very useful.
Many of these changes came from a slight change in mentality from that of reliability to one of trying to optimize the speed of data transmission. This mentality shift stemmed from the fact that ensuring reliability in a system where start and stop signals are nearly undistinguishable seemed to bring a number of difficult to solve issues and inefficiencies I had previously not considered.
-----------------------------------------------------------------------------------------------------------------
Return to Message Formatting
-----------------------------------------------------------------------------------------------------------------
On the very first day of this project, for the message format, I decided an idea that the format would include a message wrapped inside of a start and stop signal. Now however, possibly due to the rather messy way the signals interact (This is referring to the fact that a start and stop signal are triggered by the same occurrence, yet require different transmission orders to ensure that an extra bit is not accidentally read), the necessity of the stop signal was put into question.
As the start of the message, following the format made on day 1, has two bytes of commands to it, one for addresses and one for message length, the stop signal may not be needed. This is because the length of the message will be known to the receiver, due to the message length command byte. With this information the receiver will simply be able to terminate its reading of the transmission after that amount of data is received.
That simplifies the protocol to the following format:
(Start) [Message Length] [Sender/Receiver Addresses] [Data1] [Data2] ...
In fact, the start signal too is not necessarily needed. And, although it is likely to eventually bring some reliability issues, I removed that part of the message format as well. If it does not work out well however it should not be too difficult to add back in.
[Message Length] [Sender/Receiver Addresses] [Data1] [Data2] ...
-----------------------------------------------------------------------------------------------------------------
The Data Pulses
-----------------------------------------------------------------------------------------------------------------
Unlike the message format changes, the changes which will be made to the way the messages are encoded are necessary for the transition to a more expandable system. This is due to a small detail I overlooked near the start of the project.
That detail is that if all of the controllers connected to the protocol are able to both transmit and receive data, the pins which they connect to the data and clock buses to will need to be able to hold the 'off' voltage while set to an input mode. With the current 'off' voltage being LOW this is simply not possible for Arduinos. Their internal pullup resistors are connected to power!
Well, what does that mean for the signals? Flip them upside down!
While we are at it though, why not change them a little to help with efficiency?
As yesterday's second experiment showed, the Arduino is able to consistently read simultaneous pulses on the clock and data lines. The pulses used in the protocol up to this point are staggered, that simply isn't needed. Removing the delay should be able to double the speed that data is transferred at, a very substantial change. As the changes to format removed the need for start and stop signals, only the data bit transfer signals need to be made for this updated system.
Also this significantly simplifies the rules. Or rather, now a single rule:
-
When clock changes value, read bit from data line
-----------------------------------------------------------------------------------------------------------------
Plans for Future Versions
-----------------------------------------------------------------------------------------------------------------
Having a completed demo from Wednesday, next week will bring work on a version of the protocol which will be usable inside of actual projects. This means turning it into a library and moving the data transmission to a background process. Also, as I begin working on it I will implement a lot of the efficiency related changes which I found in the last few days.
Posted: Friday, 28 oct 2022