Site Development Notification and Feedback Thread

byte[]
Non-Fungible Trixie -
Verified Pegasus - Show us your gorgeous wings!
Preenhub - We all know what you were up to this evening~
An Artist Who Rocks - 100+ images under their artist tag
Artist -

Philomena Contributor
Because some object storage offerings have recently become extremely compelling, and to be honest because it’s a pain to manage millions of image files on a local server, I am once again planning a move in the application code to take images from locally hosted to cloud hosted. Because cloud object storage solutions generally don’t allow symlinks and path-based header rewriting, this will involve changing bunch of image URLs into a permanent canonical format. I haven’t decided yet what this canonical format will be, but it definitely won’t support triggering a download of the image, and it won’t support tags in the filename.
This is not the first time I have proposed scrapping the duplicated file paths. However, to make it easier to swallow this time around, if the API downloaders are insufficient or inconvenient for your purposes, I will try to address what people are actually using the download with tags in the filename feature for: downloading bulk images from your browser.

This will generate a ZIP file on the fly from a selected source and download it as a stream; it will not take up all of your memory while it’s downloading. The source could be all of your open tabs, or it could be the contents of a search query (not sure if I should include this one or not). Importantly, you can configure whether you want short or long filenames, and included metadata or no metadata.
It will also be possible to download single files, but as before, will go through a clientside script that rewrites the file name for your local machine. (This feature depends on ServiceWorker availability and might not work if you are using private browsing mode.)
I understand that this solution won’t appeal to everyone. Leave your complaints below, because while there is some working code, this downloader is still essentially in the early prototype phase and is small enough to be rewritten if there is need for some major change.
Background Pony #5126
@byte[]
How this “ObjectStorage” will play with TOR, proxies, VPNs, and IPs (unjustly) marked as “spam”? I won’t 403 you or something?
How about mass-downloader scripts that downloads gigabytes (or more) of images? It won’t ban or throttle you for “unusual” traffic?
What about hotlinking?
TexasUberAlles
Duck - If report count was a score, he'd have the biggest score
Princess of Love - Extra special version for those who participated in the Canterlot Wedding 10th anniversary event by contributing art.
Tree of Harmony - Drew someone's OC for the 2022 Community Collab
Elements of Harmony - Had an OC in the 2022 Community Collab
Non-Fungible Trixie -
Kinship Through Differences - Celebrated the 11th anniversary of MLP:FIM!
Verified Pegasus - Show us your gorgeous wings!
Preenhub - We all know what you were up to this evening~
Philomena - For helping others attend the 2021 community collab
Twinkling Balloon - Took part in the 2021 community collab.

I was honestly under the impression that being able to see an image onscreen in the first place meant that your device had already downloaded it; is the site serving thumbnails or condensed versions instead?
PUBLIQclopAccountant
Non-Fungible Trixie -
Magical Inkwell - Wrote MLP fanfiction consisting of at least around 1.5k words, and has a verified link to the platform of their choice

Pubilq Phirm
@TexasUberAlles
If you click (or tap, if you’re a phone poster) the image, it’ll download the full resolution. I learned this trick as a necessity to view details in some comics. The default is tuned to the resolution of your viewport, but you’d have to ask byte for specifics.
The Smiling Pony
Princess of Love - Extra special version for those who participated in the Canterlot Wedding 10th anniversary event by contributing art.
Tree of Harmony - Drew someone's OC for the 2022 Community Collab
Elements of Harmony - Had an OC in the 2022 Community Collab
Non-Fungible Trixie -
Thread Starter - Report site bugs here
A Really Hyper Artist - 500+ images under their artist tag
Kinship Through Differences - Celebrated the 11th anniversary of MLP:FIM!
Verified Pegasus - Show us your gorgeous wings!
Preenhub - We all know what you were up to this evening~
Philomena - For helping others attend the 2021 community collab

Somehow Omnipresent?
@TexasUberAlles
This refers to the difference between View and Download (one just opens the image, the other triggers a download prompt), plus the ability to offer an image with (some) tags in filename.
‘Save Image’ and ’Drag ‘n Drop’ type of image saving shouldn’t be affected.
The Smiling Pony
Princess of Love - Extra special version for those who participated in the Canterlot Wedding 10th anniversary event by contributing art.
Tree of Harmony - Drew someone's OC for the 2022 Community Collab
Elements of Harmony - Had an OC in the 2022 Community Collab
Non-Fungible Trixie -
Thread Starter - Report site bugs here
A Really Hyper Artist - 500+ images under their artist tag
Kinship Through Differences - Celebrated the 11th anniversary of MLP:FIM!
Verified Pegasus - Show us your gorgeous wings!
Preenhub - We all know what you were up to this evening~
Philomena - For helping others attend the 2021 community collab

Somehow Omnipresent?
@PUBLIQclopAccountant
The site creates a number of “preview thumbnails” of various sizes to best fit the current window size; it avoids always transferring the full image size when it’s not necessary at all (ie. when someone’s on a 1080 screen and the image is 4k). Clicking on it will trigger loading the full size. This change won’t affect that.
Background Pony #8FCB
@byte[]
Not having the ability to basically wget images with the tag names will definitely be a pain point for me.
Also, why don’t you do it on the server side?
I used to work on an application which served hundreds of millions of files to the “users” (other programs). Each request were internally redirected to the given object (azure blob storage), the user did not see how the item was stored. A lot of items was mapped from multiple urls to the same object. Even before the blob storage there was no such abuse of the fs what you mention. That’s what databases are for. Look up the path in that and serve the resource on whatever request URL regardless where it comes from.
Right now I’m running a smaller fan site (won’t name it) with around 500k items, each can be served on 3 different URLs, and there is only a single file for each item. Literally 1 line of inline perl in the nginx config to tell nginx what and how to remap. The generated html page contains a link which is generated by the server side code and nginx knows how to map the generated urls to the actual resource. That 1 single line of perl code even maps the objects to a simple tree structure to avoid having 500k files in a single folder (that would kill the fs and drag the iops down hard, does not matter what the backing storage is capable of). Yes, yes fs caching and stuff I know, there are reasons I don’t have them enabled/not sufficient amount.
And with this change (if there is no other option) you basically kill any other users of philomena. Not everyone want’s to have to use that specific object storage service.
Also buy your own hardware, and forget hdds. I just don’t get how a proper large nvme ssd’s millions of IOPS are a bottleneck. Especially if multiple ones are in a mirror mode where the reads can be distributed. Paying for an object storage service is more expensive. A lot more, especially considering the traffic.
byte[]
Non-Fungible Trixie -
Verified Pegasus - Show us your gorgeous wings!
Preenhub - We all know what you were up to this evening~
An Artist Who Rocks - 100+ images under their artist tag
Artist -

Philomena Contributor
Not having the ability to basically wget images with the tag names will definitely be a pain point for me.
I suspect that if you’re already at the point where you’re using wget, you can automate renaming them
Also, why don’t you do it on the server side?
Because then I can’t take advantage of CDN caching
1 line of inline perl in the nginx config to tell nginx what and how to remap
The nginx configurations I have currently are similarly small; but it breaks caching
you basically kill any other users of philomena
I already have the consideration of other philomena users in mind. They will get whatever the development environment gets, which would either be a file storage plugin or a local object storage server like minio. I would never require storage to be tied to an external service.
Paying for an object storage service is more expensive.
If you look at the links I provided, you will see that these object storage services are egress-free, and the relative cost is less than renting equivalent amounts of dedicated or block storage.
Also buy your own hardware
Great idea, I’d love to! Just one small problem: who will pay for the upfront cost of that?
PUBLIQclopAccountant
Non-Fungible Trixie -
Magical Inkwell - Wrote MLP fanfiction consisting of at least around 1.5k words, and has a verified link to the platform of their choice

Pubilq Phirm
Suggestion: downgrade hidden tags to mere spoiler tags specifically when browsing my:faves. I’ve also had hidden tags cause missing episodes from comics, but I can’t think of a good way to square that circle at the moment. However, it’s annoying when a favorite image disappears because someone added a tag you’ve hidden to it and you specifically chose to add it to your favorites collection so that you could enjoy seeing it again.
Princess Celestia
The End wasn't The End - Found a new home after the great exodus of 2012

Administrator
Devops
Also buy your own hardware
Great idea, I’d love to! Just one small problem: who will pay for the upfront cost of that?
Not to mention that kind of kills scalability. Plus, cost of colocation can be higher than what I estimate the cost of operation would be post-move to cloud. Then in 5 years when we run out of storage on that shiny server we bought, we have to drop lotsofmoney to either upgrade or replace the obsolete server, also requiring untrivial amounts of manual labor to move everything.
Also, I suspect your site doesn’t get to the point where if you don’t cache you run out of server bandwidth instantly.
byte[]
Non-Fungible Trixie -
Verified Pegasus - Show us your gorgeous wings!
Preenhub - We all know what you were up to this evening~
An Artist Who Rocks - 100+ images under their artist tag
Artist -

Philomena Contributor
Tag suggestions for uploads demo
Loads the 50 highest frequency tags for a random set of images that match the tag input query so far. Needs some tuning as the results are not currently super relevant or ordered in a sensible way, and 50 is probably too many.
The tags are all clickable and add themselves to the input when clicked. Hopefully this will lead to less barebones tagging in the future when the feature is finalized.
The button is just for demonstration purposes, you won’t actually have to click anything to load the suggestions. The suggestions will load on their own every once in a while.
Sff
Inspiration Manifestation - For Patreon supporters
Fine Arts - Two hundred uploads with a score of over a hundred (Safe/Suggestive)
Non-Fungible Trixie -
Perfect Pony Plot Provider - Uploader of 10+ images with 350 upvotes or more (Questionable/Explicit)
Cool Crow - "Caw!" An awesome tagger
Wallet After Summer Sale -
Bronze Bit -
Magnificent Metadata Maniac - #1 Assistant
Happy Derpy! -

What does it mean?!?
@byte[]
Nice to see this coming back as an official feature, the userscript for it has been broken for quite a long time
TexasUberAlles
Duck - If report count was a score, he'd have the biggest score
Princess of Love - Extra special version for those who participated in the Canterlot Wedding 10th anniversary event by contributing art.
Tree of Harmony - Drew someone's OC for the 2022 Community Collab
Elements of Harmony - Had an OC in the 2022 Community Collab
Non-Fungible Trixie -
Kinship Through Differences - Celebrated the 11th anniversary of MLP:FIM!
Verified Pegasus - Show us your gorgeous wings!
Preenhub - We all know what you were up to this evening~
Philomena - For helping others attend the 2021 community collab
Twinkling Balloon - Took part in the 2021 community collab.

Not something I would ever use, but if it helps prompt people to stop uploading images with four or five tags it’ll be worth its weight in palladium.
Interested in advertising on Derpibooru? Click here for information!
Sky Railroad Merch Shop!

Derpibooru costs over $25 a day to operate - help support us financially!

Syntax quick reference: **bold** *italic* ||hide text|| `code` __underline__ ~~strike~~ ^sup^ %sub%

Detailed syntax guide