About database downloading

[aj3423]

Hi,

Came across this discussion about intergrating database downloading in SpamBlocker.
Previously it's not recommended due to limited hardware resources, how about now? Can I integrate it now? Or is it still not recommended?

Thanks.

[haumacher]

PhoneBlock offers this download - for the service it's not optimal, but if people want it, you may give it to them...

[haumacher]

Please make sure to randomize the synchronization time to distribute the load evenly.

[aj3423]

for the service it's not optimal, but if people want it, you may give it to them...

Then it won't be added to the presets. Users will still be able to find and manually import it from the wiki, starting with a smaller user base and we'll see how it goes.

Please make sure to randomize the synchronization time to distribute the load evenly.

By default, it'll be scheduled as "periodically 23 hours", so the first sync happens when the user enables it and the subsequent sync time shifts slightly everyday..

And I'll make sure to report the number back when a reporting API is configured.

[haumacher]

I'm on my way to add incremental blocklist synchronization to the API. The current prototype is live on the test system: https://phoneblock.net/pb-test/api

Description of the mechanics can be found here: https://github.com/haumacher/phoneblock/blob/master/INTEGRATIONS.md#7-community-blocklist-download

Blocklist sync was necessary for supporting Apple devices - those do not allow live lookups during call processing (new app for iOS is being developed by independent developer, see https://phoneblock.net/pb-test/setup-iphone). However, with blocklist sync and local processing, there is no automatic give-back for maintaining the blocklist. If all users enable immediate blocking say for all numbers with at least 2 votes, there is no longer any feedback to the service after a number reaches two votes. Therefore, I'm currently not sure whether I'll include numbers with low votes to the blocklist for download in the future.

[MarkyMarkDE]

I'm on my way to add incremental blocklist synchronization to the API. The current prototype is live on the test system: https://phoneblock.net/pb-test/api

Description of the mechanics can be found here: https://github.com/haumacher/phoneblock/blob/master/INTEGRATIONS.md#7-community-blocklist-download

Blocklist sync was necessary for supporting Apple devices - those do not allow live lookups during call processing (new app for iOS is being developed by independent developer, see https://phoneblock.net/pb-test/setup-iphone). However, with blocklist sync and local processing, there is no automatic give-back for maintaining the blocklist. If all users enable immediate blocking say for all numbers with at least 2 votes, there is no longer any feedback to the service after a number reaches two votes. Therefore, I'm currently not sure whether I'll include numbers with low votes to the blocklist for download in the future.

What if you create a version for Android as usual and as you want, and a modified version for iOS. The external Dev like to modify a App for iOS, so in case of doubt, he has to load all the data and then output it to his app in such a way that Apple has no problem with it. I know why I've hated Apple since day one (Win 95) and still do! I know this all sounds very harsh, but why should what Apple does be your problem?

I'm pretty sure that you can technically distinguish where the query comes from, i.e., which OS, and then redirect it accordingly, or there are two different APIs, one for normal websites, Android, PhoneBlock app, SpamBlock app, etc., and one that is only used for Apple devices, due to their rules. Then everyone should be happy, right? 😉

Or we do it in the same way as for PhoneBlock_mobile App with a special Useragent in the request header for Apple mobile devices or something.

[haumacher]

I've to find the optimum in three dimensions:

• Reach the maximum amount of users - so Apple support is cool, if there is someone who wants to take care of those users.
• Maximum server efficiency to make it affordable to provide the service free of charge in the long term.
• Reach autonomy - have enough user contributions to maintain the blocklist - or better to have the best and most up-to-date blocklist available.

Blocklist download has two drawbacks - high server load/bandwidth requirement and reduced user contributions. Incremental sync fixes the first, but not the last.

[MarkyMarkDE]

I've to find the optimum in three dimensions:

* Reach the maximum amount of users - so Apple support is cool, if there is someone who wants to take care of those users.

* Maximum server efficiency to make it affordable to provide the service free of charge in the long term.

* Reach autonomy - have enough user contributions to maintain the blocklist - or better to have the best and most up-to-date blocklist available.

Blocklist download has two drawbacks - high server load/bandwidth requirement and reduced user contributions. Incremental sync fixes the first, but not the last.

This is mainly me trying to think along with your comment about blocklist downloads,
where you describe the tension between reach (especially iOS), server efficiency,
and the loss of feedback once numbers are blocked locally.
In particular, I found your point interesting that incremental sync helps a lot with load,
but does not really solve the feedback/autonomy issue by itself.

Very tentative thoughts – meant only as thinking aids

Please take this as very tentative brainstorming, not a proposal or recommendation.
We discussed these thoughts together; ChatGPT mainly helped to structure and phrase them in English.
I’m very much a non-expert in this area (and not a Java developer), so this is intentionally kept vague and high-level.
If anything below is obviously naive or already ruled out, feel free to ignore it completely — the intention is only to offer possible mental hints, not to suggest solutions.

---

Thinking along the three tensions you described

If I understood your comment correctly (and I may well misunderstand), the core problem space you outline revolves around balancing:

1. Maximum reach, especially including iOS / Apple constraints
2. Server efficiency and scalability
3. Preserving enough user feedback (“autonomy”) so the blocklist stays current

You also mention that while incremental blocklist sync helps a lot with server load, it does not automatically solve the feedback problem once numbers are blocked locally.

With that framing in mind, I was wondering whether the following very rough thoughts could be useful purely as thinking prompts.

---

1. Would it make sense to distinguish between a very small “core” list and everything else?

This may be completely off, but I was wondering whether:

• a very small and very conservative core list (only extremely high-confidence numbers) could be downloaded/synced, primarily to support iOS,
• while everything else (new, uncertain, long-tail numbers) remains server-side and is handled via lookup where the platform allows it.

My naive intuition is that this might help with reach (especially iOS) and efficiency, while delaying the point at which feedback is cut off — but this might already conflict with other constraints you’ve considered.

---

2. Could iOS still contribute some signal after a call?

I understand that iOS heavily restricts what can happen during the call itself.
I was just wondering whether, after a call has ended (e.g. via call logs or notification-related contexts), it might still be possible to report something like “this number was seen” or “this number was blocked locally”.

This obviously wouldn’t help with immediate blocking, but maybe it could preserve a minimal feedback loop — unless iOS restrictions make even this impractical.

---

3. Is there any way for locally blocked numbers to avoid becoming completely invisible?

One point that stood out to me in your explanation is how downloaded blocklists can effectively silence feedback once numbers are blocked locally.

Very tentatively, could there be something like:

• extremely minimal, batched “seen/blocked” signals,
• sent with randomized delay,
• optional and privacy-conscious,

just to avoid numbers becoming entirely invisible once they enter a local list?

This may well be a bad idea or too expensive — I’m mostly trying to understand the design space around this trade-off.

---

4. Incremental sync plus chunking as a purely conceptual scaling thought?

Incremental sync already sounds like a strong step towards solving the efficiency side.
I was wondering, purely conceptually, whether ideas like chunking by prefixes/regions, aggressive HTTP caching, or CDN-backed delivery of static artifacts are things you’ve already evaluated or ruled out.

This is not meant as a suggestion, just me trying to follow the scaling implications of the approach you described.

---

5. Soft handling instead of hard blocking for uncertain cases?

Another very rough thought: instead of pushing borderline numbers into a hard blocklist early, maybe there’s room for softer handling (labeling, silencing, warnings), depending on platform support.

That might preserve user comfort while still allowing feedback to accumulate before numbers become hard-blocked everywhere — but this might already conflict with UX or simplicity goals.

---

Why I thought this might touch all three dimensions (very tentatively)

• Reach: a tiny conservative local list could help on iOS
• Efficiency: smaller downloads and incremental updates reduce load
• Feedback: even weak, delayed signals might be better than none

But again, this could be entirely wrong — the goal here is only to offer some thinking prompts while trying to follow the three competing constraints you outlined.

---

If you read this far: thanks — and please feel free to treat all of the above as noise if it doesn’t fit your mental model at all 🙂

[haumacher]

Cool, yes, it fits my mental model exactly :-)

Only including number with high confidence into the blocklist for download is a valid option. Of cause, this is giving the users on OSes that allow to only use cached blocklists the impression that the blocklist is not as accurate as it could be - since calls can pass that would have been blocked otherwise.

Distributing blocklist and blocklist increments to CDN space for mass downoad is one option I'm considering. The incremental sync protocol is designed with that option in mind.

As I understood (#199 (comment)), @aj3423 is providing functionality to provide some feedback, if a call created a hit in the blocklist. This is critical for keeping the blocklist alive. But however, as I understand - this is not possible on iOS - without the user itself taking some action (but that only what I've heard, I not even think about developing for iOS).

So yes - when using the blocklist sync option, keep in mind that I could limit the download to only numbers with high confidence in the future, if I think it is necessary for keeping the service alive. Currently, the main goal is to increase the number of users. Therefore, the full blocklist is available for download and sync now and for the foreseeable future.

To be future-prove, it would be a good option when enabling blocklist download in an app to also add an option with which a user can enable live lookup if there was no hit in the downloaded blocklist. Maybe, I should add this to the integration guide.

[MarkyMarkDE]

Cool, yes, it fits my mental model exactly :-)

Only including number with high confidence into the blocklist for download is a valid option. Of cause, this is giving the users on OSes that allow to only use cached blocklists the impression that the blocklist is not as accurate as it could be - since calls can pass that would have been blocked otherwise.

Distributing blocklist and blocklist increments to CDN space for mass downoad is one option I'm considering. The incremental sync protocol is designed with that option in mind.

As I understood (#199 (comment)), @aj3423 is providing functionality to provide some feedback, if a call created a hit in the blocklist. This is critical for keeping the blocklist alive. But however, as I understand - this is not possible on iOS - without the user itself taking some action (but that only what I've heard, I not even think about developing for iOS).

So yes - when using the blocklist sync option, keep in mind that I could limit the download to only numbers with high confidence in the future, if I think it is necessary for keeping the service alive. Currently, the main goal is to increase the number of users. Therefore, the full blocklist is available for download and sync now and for the foreseeable future.

To be future-prove, it would be a good option when enabling blocklist download in an app to also add an option with which a user can enable live lookup if there was no hit in the downloaded blocklist. Maybe, I should add this to the integration guide.

Thanks a lot for the detailed reply — that’s really encouraging to read 🙂

It’s especially helpful to hear that this fits your mental model. I think that already answers more than half of my (very tentative) questions.

Your point about prioritizing user growth right now makes total sense to me. Having the full blocklist available for download initially seems like the right trade-off at this stage, even if it means accepting the feedback limitations for now. The idea of potentially tightening this later to high-confidence numbers only feels like a very reasonable future lever to keep the service alive.

Regarding feedback: thanks for clarifying the situation around iOS. My understanding was similar — that without explicit user interaction, feedback signals are very limited there. I also saw the comment you referenced in #199 (from @aj3423), and the ability to report hits when a call matches the blocklist indeed sounds critical for keeping the list “alive” on platforms that allow it. Unfortunately, iOS seems to be the odd one out here.

Your last point about future-proofing integrations resonated with me a lot:

enabling blocklist download, but also allowing live lookup when there is no local hit

From a purely conceptual (and very naive) perspective, that sounds like a great way to combine comfort, reach, and long-term data quality — especially if it’s clearly documented in the integration guide so app developers are nudged in that direction early on.

Thanks again for taking the time to explain your thinking so clearly. For me, this discussion was already very helpful in understanding the trade-offs you’re dealing with.

[aj3423]

Two questions about the parameters:

• minVotes
  It's missing from the doc but still works, is it deprecated?
• since
  What happens if it's queried with an old version number? Say, after restoring from a backup that was created a month ago.

[aj3423]

done: https://github.com/aj3423/SpamBlocker/actions/runs/21390596497

One question, from the doc:

Use randomized intervals (e.g., 23-25 hours) to distribute server load evenly throughout the day.

So the workflow is currently set to run every 23 hours.

However, on the API page:

Incremental synchronization (with 'since' parameter) should be performed at most once per day.

Will running it every 23 hours cause any issues?

[haumacher]

Will running it every 23 hours cause any issues?

No, once per day is a general guideline. However, a better distribution would result from 23.5 + Math.random() hours.

[aj3423]

a better distribution would result from 23.5 + Math.random() hours.

I understand that's necessary for downloading the full list. But with the new incremental update, does "syncing once a day" vs "twice a day" still make a significant difference?
I think "syncing twice a day" is actually better, the server sends less data per request, which helps spread out the load more evenly.

[haumacher]

Currently, the server decides when to make new versions of the blocklist available. So syncing more often makes no sense, since no new data will be available. Let's start with aprox. once a day - and see how large the changes are.

[aj3423]

Okay, I'll change it to 23:50, looks "random" enough.

[haumacher]

The incremental sync is now live (version 1.9.1).

[aj3423]

Got it, thanks.
New release is out.

[Joduai]

great to know about both updates
just tried out how things work, added couple spam numbers, checked on pb website if they show up, couple mins later got a spamcall silenced using database setting
seem to work like a charm 👌