For a few months, I was working on a syncing system.
There were a few local operations that I wanted to sync with a remote server.
The first thing I tried to keep the system simple, It was:
- listen every operation
- call the api to sync it
- and wait for the response.
This approach wasn't usable at all:
-
it caused a lot of network requests,
-
And it caused a lot of latency delay in the UI.
And since the network isn't always reliable, if it failed in the middle of request, the data could be lost permanently.
After facing these problems, I rethrought the system and I came up with this new approach:
I decided to store the data locally using sqlite or indexedDb then on every event:
- save it to local storage
- periodically (e.g every 5 seconds) sync with the server
- handle conflicts during sync
This was better but still not satisifed.
It still depended on a fixed interval to sync data which I didn't feel efficient.
Then I try to use a queue system.
the queue had:
- constraint (size limit)
- retry mechanism (for failed syncs)
- exponential backoff (to avoid overwhelming the server)
Now the flow became:
- push data into the queue
- check the queue size
- if the size exceeds the limit, trigger a sync
- if sync fails, retry with exponential backoff
This made the system more controlled and prediectable
And this is essentially the idea used by figma, notion, and other real time collaboration systems.
The ui feels instant becuase it hides network latency data isn't lost because its stored locally. and syncing happens in a more reliable and way more efficent way, then I can explain becuase I don't work there and not handling that much of chaos.
One more thing:
No matter how fast your server is, even if you write it in some black magic assembly, network latency will always exists.
you can't change the laws of physics.