Shop Mobile More Submit  Join Login
Last year, we released the deviantART and Sta.sh APIs, our first official support for third-party app integrations. Eager developers have asked for many new APIs since then, anxious to step away from page-scraping and toward officially supported integration. We're eager for that too! :eager: 

Today we are happy to announce three new Sta.sh APIs which give developers greater access to their users' Sta.shes!

---

The New Sta.sh APIs

  • Fetch Submission Media - Now developers can request the filesize, dimensions and URL for the original media associated with any Sta.sh submission.

  • Fetch Submission or Folder Metadata - Apps can now get all of a submission's dirty details (keywords, artist comments, thumbnail URLs, etc). It can also be used with Sta.sh folders for greater organization capabilities!

  • List Folders and Submissions - My personal favorite and the workhorse of this release: this API call gives developers full read access to a user's Sta.sh in an intelligent way. We'll look more at the tech behind this call below.
--- 

The Challenges of a Delta

List Folders and Submissions (aka /delta), the most powerful call in this release, uses an incremental approach to allow apps to retrieve and store the current state of a user's Sta.sh in the most efficient way possible. By using the current state, developers and apps can recreate the Sta.sh experience on any device with any interface they desire. Let's look at what it does, how it works, and some of the challenges that dT faced during development.

Users can have up to 10GB of deviation submissions in their Sta.sh - literally thousands of 1Mb submissions. Each submission belongs to a folder, and folders can have many submissions. For an application to represent this to the user, it needs to know everything about the submissions as well as the containing folders. Because submissions can be updated, published, deleted, and moved between folders, an application needs to constantly check for the data.

Challenge 1: How can a third party application check the status of thousands of deviation submissions and Sta.sh folders repeatedly without drastically increasing server load and lagging the app?

Challenge.. Accepted: The solution was inspired by Dropbox's /delta API (though it's fundamentals can be seen throughout the world of computer science). The first time an application loads a user's Sta.sh state, we give it everything. All of the folders, all of the submissions. That kind of transmission could be megabytes, so we chunk it up into result sets of roughly 120 submissions. The application makes as many calls as necessary to retrieve the full list, and stores all of this information locally; when it completes, it will contain the exact state of that user's Sta.sh.

We also give the app a unique cursor which represents that exact state of that user's Sta.sh. After the user interacts with Sta.sh (submitting new deviations, modifying existing ones, publishing and deleting, moving deviations between folders), the application can send us the cursor and we'll give it everything that has changed. Checking for updates becomes a very fast, repeatable process for the application.

Challenge 2: How do we build the list of modified submissions and folders?

Challenge.. Accepted: Let's reconsider the data challenge above: 10GB of deviations = thousands of submissions = tons of data. Per user. Storing the exact state of a deviation at each step in the process would.. well, it'd be like striking the immovable object with the unstoppable force. Chuck Norris would disappear from existence. :confused:

Aiming to save ol' Chuck, we implemented a circular log table to track each user action in Sta.sh; we can store as many as one hundred and sixty thousand user actions. Anything above that wraps the table and automatically deletes old entries.

Each cursor corresponds to a specific log entry, so we can quickly compile a list of modified items since that cursor. Note that we do not record the state of the item in the log - we just record the fact that it changed in some way. This lets us fetch data for a small list on request, making the delta call cheap and fast.

Challenge 3: What happens when the circular table rolls over and old entries are lost? What if my application's last cursor becomes invalid? What if a submission is added and deleted between delta calls?

Challenge.. Accepted: The first two questions are examples of edge conditions where an incremental approach breaks down. To compensate, we have a special flag called reset that tells the application it needs to throw away its stored data and start over.

Because we only log the change in items, the list of those changes is likely to contain useless transactions like an submission that was added and deleted between delta calls. Applications don't care about this kind of entry - it won't appear in the app anywhere, doesn't exist anymore, and is simply confusing to see in the delta list. We applied an extra layer of filtering on the changelog to ensure that the application receives relevant, useful updates.

---

That's cool and all.. but where is the API for [awesome feature X]?

As kouiskas pointed out in last year's journal, adding new APIs is a time-consuming process; once we release an API, we want to support it forever. This limits our choices to APIs that are stable, scalable, and compatible with our long-term development goals. 

There have been numerous requests for great APIs over the last year; unfortunately, we can't build them all. Many would rely on infrastructure that will be refactored (e.g., Message Center) or that is in the process of being refactored (e.g., Search). 

Does this mean you should stop requesting those APIs? Not at all. Keep the requests coming! If you're an app developer and really, really want to see a specific new API call, here are some things that will help us review your request:
  1. Give us some stats. 
    If you have a successful app that is being used daily by deviants worldwide, growing in popularity and proving a benefit to the community, throw some numbers our way. Although popularity doesn't affect what is and isn't possible, it can be an important factor in deciding where to concentrate our efforts for the most benefit to most users.

  2. Show us what your app will do with the call. 
    Most of you already have very successful dA apps in the desktop and mobile world. We realize that there is a lot of DiFi hacking and page scraping necessary to make your app do what you want it to do. Show us how the new API will reduce your reliance on these methods. Include screen captures if you can, we like to look at shiny XHR logs and fail messages.

  3. Be specific and technical.
    We're developers, just like you. If you see a need for an official API, take the time to help us understand exactly what you want from the API (request parameters and result objects). This doesn't mean that we will honor your spec request exactly (we do what we can with the infrastructure we have), but it will help us understand your need more clearly.
Add a Comment:
 
:iconvocable:
Vocable Featured By Owner Sep 3, 2012   Writer
Wooh, this looks awesome. I do love that you document what you did and tell us about it. :D

<3 for transparency but also awesome storytelling
Reply
:iconbaronbeandip:
baronbeandip Featured By Owner Aug 28, 2012
I'm glad to finally see some new OAuth2 API calls! This is just what I needed to improve devStash (for Android). My user base will surely love the new features.
Reply
:iconzombiecoder:
ZombieCoder Featured By Owner Aug 28, 2012
I look forward to seeing what you do with it! :excited:
Reply
:iconvanilla-vanilla:
vanilla-vanilla Featured By Owner Aug 28, 2012
Sigh... I'm still waiting for just basic site functionality like... Downloading all of the notes I've sent/received, so I can find stuff, and also clear out my in/out boxes...! :-)
Reply
:iconzombiecoder:
ZombieCoder Featured By Owner Aug 28, 2012
Heh, aren't we all. Check out my replies to other comments about Message Center (the functionality you're describing), or $banks's post about it here: [link]

Basically that functionality is old and due for a refactor, and we can't support an API on it yet. We will be refactoring it in the future, and when that happens, there will be an API for it.. but it is not going to happen in the near future.
Reply
:iconphotofroggy:
photofroggy Featured By Owner Aug 28, 2012  Hobbyist Artist
I also think it's worth noting that if an API is designed well, then changes to the underlying implementation and integration should not be much of a concern.

If you have already started work on designing API calls, why not ask for feedback on your ideas? Would be far better than radio silence.
Reply
:iconrandomduck:
randomduck Featured By Owner Aug 28, 2012
Thats is correct. But its also true only if just the implementations changes, not the functionality in itself. For example, with Message Center - the inbox/email paradigm of MC is poorly suited for making a modern mobile app, where feed-like system would be much preferable. We cant design such API in absence of the system itself :)

Also, to add to $ZombieCoder's rationale for the future APIs that we might be considering and building. We'd be far more likely to be interested and motivated to work on the APIs to support original and creative apps that enhance and compliment the user experience, and not the ones that clone the web site or work around some quirk, or even worst, facilitate spamming. (sorry if I offend someone)

Let us know what you guys think of building apps that add value to core features we have. Just to give an example - how about a mobile location-aware devmeet app? Not simply a gateway to group devmeet module, the app doing way, way more - scheduling meets, finding one in your country, doing checkins, making and sharing photos taken there, eazily watching attendees they just met, Did anyone have that idea already? I bet you did. Things of that nature rather than requests to add oauth call for add/remove watchers and faves should be way more interesting to all of us.
Reply
:iconphotofroggy:
photofroggy Featured By Owner Aug 30, 2012  Hobbyist Artist
If by "feed-like system" you mean turning the entire message centre into a facebook wall or twitter feed style thing, then I must say; that is a terrible idea. Different sections of the message centre would, and already do, work well as feeds. Bundling everything together would make things way too cluttered, and an absolute nightmare to work with.

The inbox/emails paradigm is poorly suited for mobile applications that deal with feeds. There are "modern mobile apps" for managing emails. Your point here doesn't make much sense to me. You're making me worry that people working at dA think everything needs to be more like twitter and facebook. Just because one formula works doesn't mean others don't. Also keep in mind that application developers would make their applications appropriate for the problem they're trying to solve. Taking a completely wrong approach would be absolute lunacy.

Designing an API can help in designing the system itself. Ideally you need something that is simple but still offers greater control in some way. When actually working on the system itself, modify your API designs as appropriate. Let the two things bounce off each other. When designing the API throw your ideas out to people and discuss it. Feedback is always useful, and if it's a public API it is definitely worth running it by the people who will be using it.

People want to make original and creative apps, but some underlying functionality is always needed. For your devmeet idea, you would need an API for the devmeet module and various other things.

"clone the website" last time I checked, the mobile version of deviantART was terrible. Kind of looked as though it was designed to look and feel like an "app", failed somewhat, and tried to tackle every feature in one package. The scale of dA is too big to be in one application on mobile devices.

So, with the mobile website not really being a viable option, what mobile applications have dA released? None at all? How can you compliment a user experience that does not exist? It seems obvious that people would want to fill that huge gap. I would hazard a guess and say that when the core functionality of the site is supported in some way on mobile devices, then the "creative" stuff will come after. But your argument here is kind of invalid; if you're not motivated enough to support mobile platforms, how do you expect to motivate others to be creative? The ball is, and has been, in your court for a long time.

It is also worth noting that people want to be able to use dA away from home. That doesn't necessarily mean having a devmeet abstraction. Sure, it would be cool, and it would make things easier if it were possible to do more in the first place.

Oh, and, any application or program code that allows you to autonomously submit content to a website is facilitating spamming. A way to prevent that sort of thing, at least somewhat, would be to limit the number of API calls an application can make, or just limit the number of API calls that result in content being created.
Reply
:iconrandomduck:
randomduck Featured By Owner Aug 30, 2012
You are putting words in my mouth. Nowhere did I say facebook or twitter. feed is a far bigger and older a concept and not restricted to what they do.

As for mobile site, stay put, changes are coming. and soon

API access to web application core is a decision at a product development level. We can speak here only on the technical one. I'd note that api-rich ones like the facebooks of the world are very heavily sandboxed and would not let you clone the entirety of the functionality through those.
Reply
:iconphotofroggy:
photofroggy Featured By Owner Sep 1, 2012  Hobbyist Artist
I was not putting words in your mouth. I said "If", and voiced my opinion on that possibility.

Mobile sites seldom work very well, unless it's for something simple like a news feed. I think you would be better off developing a mobile application for dA rather than a mobile website.

Regardless of where the decisions are made, surely you guys talk to each other? Those particular decisions seem like they should be made by, or at least discussed with, the more technical people, and those who would be working on and with the API.

One thing that is particularly frustrating; you ask for suggestions and feedback, but keep the discussions very one sided. Responses like "stay put" and "it's an old feature" aren't particularly great responses. Why not discuss things more openly? If it's going to be released as a public API then where is the problem in discussing it?

Speaking of old features needing a refactor: wouldn't it be better to do that before coming up with even more features that you don't have the man power to support side by side? The chatrooms are long overdue for some lovin', as is the message centre.
Reply
:iconphotofroggy:
photofroggy Featured By Owner Aug 28, 2012  Hobbyist Artist
I think the documentation pages need improving.

If the API grows over time, the current documentation pages will become cluttered. The Sta.sh documentation already feels a bit huge, and the navigation doesn't make sense to me.

Using an asterisks (*) to indicate there is more than one API call in a set seems odd.

The pages just don't seem too usable as actual documentation. I'm not exactly better with writing documentation and such, but presentation is pretty important. When skimming over things it feels as though you guys ignored every popular method of presenting documentation and decided to do things in a pretty disorganised and unstructured manner, but that's mainly at a glance.

Would be great to have official documentation on other aspects of the website, too. #Botdom et al can only do so much.

And I've been saying for ages that a message centre API is needed. Badly.
Reply
:iconzombiecoder:
ZombieCoder Featured By Owner Aug 28, 2012
I can't agree more that the API docs need improvement. The DiFi docs that the community has put together is better than ours! I've got my fingers crossed that we'll be able to spend time on a quick cleanup of those docs, if not a full overhaul.

What exactly are you describing in regards to "documentation on other aspects of the website"?

See my response to =logeg regarding the MC API.
Reply
:iconphotofroggy:
photofroggy Featured By Owner Aug 30, 2012  Hobbyist Artist
Well, DiFi would be an example of something that could do with some official documentation. While there are third party docs for that, they are still incomplete, and there's only so much we can know from reverse engineering. I have been rebuffed on the matter before because DiFi isn't meant to be a public API, which is fair enough, but it's still interesting to know about it.

The protocol for the chat server has been extensively documented over the years, but the documentation is probably still incomplete. Again, there's only so much we can know from reverse engineering.

Can't really think of anything else, but they are pretty big things. Aside from that it would just be nice to see something which you guys can use to explain and show off your work. Sure, the live product is always nice, but doesn't really translate the technical side of things very well. I suppose the narratives that have appeared on #dt have been interesting, and actually act as good documentation in some aspect. The bad thing about this is how infrequent such narratives are. But, yeah, time constraints etc.
Reply
:iconlogeg:
logeg Featured By Owner Aug 27, 2012  Hobbyist General Artist
As I explained in detail in a letter to your team (and I have a reply pending from $randomduck), there's a real need for Message Center API.

There are at least 2 widespread applications that poll known DiFi interfaces to monitor user and group inboxes, #deviantAnywhere and #dANotifier.
I represent the latter, and as for stats - Chrome Web Store reports 9k+ active installations.

Problem is, DiFi calls are extremely ineffective for what is needed to be done (all explained in detail in the email) and, of course, are your internal and unsupported APIs, prone to break.
Reply
:iconrandomduck:
randomduck Featured By Owner Aug 28, 2012
sorry, I dropped the ball on answering that email, but I just saw that $ZombieCoder did and did it well. hope that helps.
Reply
:iconlogeg:
logeg Featured By Owner Aug 29, 2012  Hobbyist General Artist
No worries at all! I can imagine how busy you are.
Reply
:iconzombiecoder:
ZombieCoder Featured By Owner Aug 28, 2012
I'm sorry we haven't responded more thoroughly to you yet! The email you sent is very thorough. How $randomduck answers any emails at all is beyond me, the guy has about ten billion things to keep track of each day. I'll review and respond to your email today, though it might not be the answers you are looking for ;)

The MC API is hands down the most requested API. It's also a system that is not easily integrated with, due for refactoring, and old. =photofroggy is correct saying that a well designed API can work regardless of the foundation it is built on, but that doesn't change the fact that we're limited in what we can do with our time and resources.

If we implement an API for Message Center now, we'll have to support it through the refactor process, adding one more point of error to an extremely complex system. It would delay our development of a new MC and take resources from other maintenance tasks.

Our official policy on new APIs is this (straight from the duck's mouth): there will be no APIs for legacy features/systems. Our goal is to build APIs as we roll out new features and systems, but that is still constrained by time and resources.

I'm not one to dangle a carrot: if MC ends up having an API, it will not be in the near future. :(
Reply
:iconlogeg:
logeg Featured By Owner Aug 28, 2012  Hobbyist General Artist
I got that much from $randomduck's first e-mail, and I completely agree with the policy!

My concerns are to efficiently use (without hope of long-term support) the current architecture and discuss future possibilities, not "Y U NO API" right now :) Looking forward to discussing it further.
Reply
:icondivinityarcane:
DivinityArcane Featured By Owner Aug 27, 2012  Professional General Artist
I gotta agree, dA needs some major additions to the site API, more-so than Sta.sh. While Sta.sh is cool and all, we want to work with the main site! We need things like an API to get the JSON data of a userpage, etc. And we're tired of relying on DiFi which is shittily done. :iconfixitplz:
Reply
:iconzombiecoder:
ZombieCoder Featured By Owner Aug 28, 2012
We're adding APIs for new features and systems only at this time. Legacy systems (which is the majority of the main site) will not be receiving APIs unless there is a refactoring opportunity to do so.

When you say "the JSON data of a userpage" - do you mean all of the page modules etc? Or just the info about the user?
Reply
:icondivinityarcane:
DivinityArcane Featured By Owner Aug 28, 2012  Professional General Artist
Well, I suppose that's okay and all. The most profitable aspect of dA at the moment is in fact the newer features like Sta.sh. It would just be nice for some of the "oldschool" things to receive updates, like dAmn, for instance.

Well, not so much the modules, the information. The majority of people who've talked about the need for such an "API" were talking about user info, though things like journal content, etc, would be very nice (and also assist in the development of more third-party dA tools, since HTML scraping is rather slow on most common networks.) I realize that there is, in a way, already a way to get deviation information, but a better JSON based "API" would be nice. This also applies to things like galleries (and scraps), folders, etc, as well as groups (information, galleries, journals, ...)


Out of curiosity, what are the thoughts of dA staff/developers in regards to things like dAmn? I was actually considering rewriting the protocol for it and making it open source for dA to pick at, since you never know what could be useful.. But that's not really anything major.
Reply
:iconzombiecoder:
ZombieCoder Featured By Owner Aug 28, 2012
Some of that information may be able to be made available via OAuth2, some of it won't. There's a strange fine line between what is legacy and what is newer development.. if you want to champion this request, collect info from the interested parties and shoot a sample JSON result object to developers at deviantART.com. I'll review it and let you know if anything can be done with the need.

Re: dAmn, it's something that is fairly integrated with our internal systems, so we wouldn't be able to work with an open source port.
Reply
:icondivinityarcane:
DivinityArcane Featured By Owner Aug 28, 2012  Professional General Artist
Sure thing, I'll look into writing something up. Thanks for the response.
Reply
:iconparallellogic:
parallellogic Featured By Owner Aug 27, 2012
~Chuck Norris would disappear from existence
:roll:

~stable, scalable, and compatible with our long-term development goals
That's pretty awesome to hear. The site changes so much in the long term, it's good to know that we may eventually have something solid to base code on.

~If you're an app developer
Do bot developers count too? I've got my own behemoth of code interacting with the message center (and the Difi hacking and page source scraping rings very true), but it sounds like it's somewhat tangential to what you're talking about, at least in the short term. I've also been listening to the questions I've gotten on my own work and it seems other users are interested in setting up small bots as well (for managing sections of group profiles or replying to repetitive questions on their profile pages). I'd have to really go through and document my work if you guys were seriously thinking of creating an API for the message center and related message center components.

A couple things off the top of my head - it'd be really great to be able to load a 100 badge transactions at a time rather than the standard 5 [link] . When I run stuff like [link] , it takes an eternity to catalog users who have above a few thousand badges.

Also, with the username change capability - it would be useful to be able to provide a current name and see all the old names that the user has previously held. I try to minimize my bandwidth usage for storing badge transactions in a file - but since usernames can now change, I have hundreds of thousands of usernames that may, or may not, be different. When I collect just the new badge activity and compare it to the old, I have the a select few new names on one hand and the old names on the other, but I would have to do a metric ton of page calls on the old names to find the ones that have changed (since the old names reference the new ones, but the new ones don't point to the old ones).
Reply
:iconzombiecoder:
ZombieCoder Featured By Owner Aug 28, 2012
Bot developers get no love!! :( Really though, we can't officially support any extension or page-scraping work.. those endpoints are officially for internal use only. Piggybacking on our systems can be done (as you and everyone else have been doing for years), but it is unsupported and something like increasing the limit of the badge transaction call is out of our hands.

Check out my reply to =logeg regarding MC - the short answer is "not in the near future". In general, we're not adding APIs for legacy systems, only new development.

I'll look into the possibility of adding username rename information, but it'd need to go over OAuth2. If that's something that can work for you, shoot a quick spec suggestion / request to developers at deviantART.com.
Reply
:iconparallellogic:
parallellogic Featured By Owner Aug 28, 2012
~but it is unsupported and something like increasing the limit of the badge transaction call is out of our hands
I guess my main concern is that I do various scripting work that puts my bandwidth usage well outside the range of a typical user (at least in terms of number of page calls, I don't load images). My fear is that my work will be terminated (the bot account will be banned). Are there any bounds dA is looking to enforce on bot (or other program) operators that I should be aware of (bandwidth or otherwise)? I believe all my work is within the dA ToS, should I be concerned about my approach?

~as you and everyone else have been doing for years
You mentioned other users are working directly with dA's message center, could you direct me to them? I'd be curious to exchange notes and ideas with others doing similar work (I'll be looking through the other comments here to collect information as well)

~In general, we're not adding APIs for legacy systems, only new development.
So would an API for the points system be entirely out of the question then? I mean, I can't really see you guys rehashing that significantly, so I doubt it would be covered under the new policy. Idk, is the dA staff alright with developers using a mixed Difi and OAuth approach for the long term? I'm fine with the risk of working with an unsupported network (nice thing about a bot over an app is I don't care too much about changes since I'm the only one affected), I just want to make sure I'm not crossing any lines when I do though.

~I'll look into the possibility of adding username rename information
:thanks: Thank you, that would be greatly appreciated.

~If that's something that can work for you, shoot a quick spec suggestion / request to developers at deviantART.com
#dt?
Reply
:iconrandomduck:
randomduck Featured By Owner Aug 28, 2012
we Chuck Norris'ed many, many misbehaving bots. I dont believe we blackholed any of yours yet :) And yeah, stay within the bounds, and you'll be fine. As for username history, I dont believe we'd be making that API - this kind of history is semi-private and a reason for feature to exist
Reply
:iconparallellogic:
parallellogic Featured By Owner Aug 28, 2012
I see spammers from time-to-time trying to throw email addresses out (~liibaby123 and the like). You guys have done a remarkable job on nixing spam messages though - do you guys run into many other kinds of bots? I know you've run into a couple account creation bots (mentioned a few I came across in [link] ), but I'm not really aware of what other kinds of bots you strictly disallow/pursue. My stuff only interacts with users that visit the bot - I figure so long as I'm not making first contact, I can avoid the spamming distinction

~I dont believe we blackholed any of yours yet
I just have the one for now, and it's a bit counter-culture to dA's mind-set (which is the part that worries me, I've put a sizable amount of time into the code base and would hate to see it vanish overnight), but I'm fairly confident I'm still within the ToS. If you guys change the ToS though and/or close the account, I'll probably write a paper on the operations, I meant it mostly as an experiment, so I'd be inclined to share my thoughts and findings which would hopefully aid in developer operations going forward.

That's unfortunate. In retrospect I'm not sure I could have interfaced with it anyway - I'm running solo scripts and a bot, not apps, so I'm not sure I'd qualify for the OAuth certification.
Reply
:iconzombiecoder:
ZombieCoder Featured By Owner Aug 28, 2012
~~ ...should I be concerned about my approach?

We discourage any kind of scripting, automation, or extension that performs dA actions outside of the dA interface.

With that said, you are not likely to get shut down if you're working within the ToS. We'll ban if your bots actions are causing a negative / harmful experience for other users or causing undue server strain. Sequential calls to the badge transaction method - note sequential, don't start DDOSing us - should not get you in trouble. If you're worried about causing problems, throw a 100-200ms delay between script calls (although that would destroy your execution time).

~~ ..other users are working directly with dA's message center, could you direct me to them?
[link]

~~ adding username rename information .. #dt?
Email us the spec at developers at deviantART.com ;)
Reply
Add a Comment:
 
×

:iconzombiecoder: More from ZombieCoder


Featured in Collections

News I Support by namenotrequired

JOURNALS AND NEWS by Elandria

dA Related by Arichy


More from DeviantArt



Details

Submitted on
August 27, 2012
Submitted with
Sta.sh Writer
Link
Thumb

Stats

Views
14,072 (1 today)
Favourites
11 (who?)
Comments
29
×