WEBVTT

00:00:00.080 --> 00:00:03.920
There's kind of a trend,
if you wanna call it like

00:00:03.940 --> 00:00:07.160
this, or definitely something we see

00:00:07.560 --> 00:00:11.510
become more common over the last one
or two years, which is

00:00:11.510 --> 00:00:14.660
not really surprising, but nonetheless,
something I wanna talk

00:00:14.720 --> 00:00:18.500
about. The trend of APIs of

00:00:18.560 --> 00:00:22.280
websites or web services becoming more and

00:00:22.360 --> 00:00:26.100
more locked down or private or more

00:00:26.140 --> 00:00:28.730
expensive to use,
whatever you wanna call it.

00:00:28.780 --> 00:00:32.360
And the most recent example, which
is the reason why I'm creating this video

00:00:32.400 --> 00:00:35.660
here, is the Reddit API,

00:00:35.740 --> 00:00:39.720
because, uh, two days ago,
there's been a post by the,

00:00:39.740 --> 00:00:43.480
the Reddit team in the Reddit development
forum where they

00:00:43.560 --> 00:00:47.379
essentially announced
that the usage of their API now

00:00:47.460 --> 00:00:50.540
needs approval. So there is an approval

00:00:50.900 --> 00:00:54.400
process for using the API for

00:00:54.460 --> 00:00:58.080
responsible use to support responsible

00:00:58.100 --> 00:01:01.980
builders, and I'll get back to that
and what that means, but it's

00:01:02.000 --> 00:01:05.679
kind of in line,
if you wanna call it like this, to what

00:01:05.740 --> 00:01:09.700
Twitter, X, did two years ago already.

00:01:09.720 --> 00:01:12.980
They made their API really

00:01:13.340 --> 00:01:16.820
expensive to use, at least at scale.

00:01:16.860 --> 00:01:20.740
So if you wanna interact with Twitter,
with X, programmatically,

00:01:20.800 --> 00:01:24.600
if you wanna build yet another social
media scheduling tool and you wanna

00:01:24.620 --> 00:01:28.420
support X, well, that could get expensive

00:01:28.480 --> 00:01:32.280
depending on how you build it, uh,
because the, the free

00:01:32.400 --> 00:01:36.000
usage is quite limited. You can,
for example, read

00:01:36.340 --> 00:01:40.240
100 posts per month and write 500

00:01:40.320 --> 00:01:43.300
posts per month,
which might be more than enough

00:01:44.020 --> 00:01:47.220
for your own little tool
that you're building for

00:01:47.320 --> 00:01:51.200
yourself, but if you
are building a SaaS product

00:01:51.240 --> 00:01:55.000
on top of the X API,
that will not suffice, so you'll have to

00:01:55.040 --> 00:01:58.900
pay,
but not even the basic tier might be

00:01:58.940 --> 00:02:02.520
case. It might be, it might not be.
And the pro tier might

00:02:02.760 --> 00:02:06.740
still not be enough. Now chances are,
it may be enough, but it

00:02:06.840 --> 00:02:10.538
is also, uh, quite expensive.

00:02:10.600 --> 00:02:14.430
And now for Reddit, there, as I

00:02:14.480 --> 00:02:18.240
mentioned, i- it's not about paying
or about, uh, the

00:02:18.300 --> 00:02:22.080
price they're asking,
but it's about an approval

00:02:22.120 --> 00:02:26.060
process so that not every application can
start using their

00:02:26.100 --> 00:02:30.040
API. And the question of course is,
why are companies

00:02:30.180 --> 00:02:34.160
doing that? Well, there
are a couple of reasons and

00:02:34.220 --> 00:02:37.960
one big important reason. Obviously,
you could say, why would they

00:02:38.100 --> 00:02:41.720
not do it?
Why would they give you access to their

00:02:41.880 --> 00:02:44.820
free? And you could argue, well,
because in the

00:02:44.880 --> 00:02:48.860
past, before AI, they may have

00:02:48.900 --> 00:02:52.540
benefited from doing so. Because
if people can build

00:02:52.580 --> 00:02:55.940
products on top of, let's say, X,

00:02:56.080 --> 00:02:59.600
if I can build a social media scheduling

00:02:59.680 --> 00:03:03.200
application,
that might be in Twitter's and X's

00:03:03.360 --> 00:03:07.090
interest because more posts on X could

00:03:07.140 --> 00:03:10.460
mean more engagement, uh,
more people reading and

00:03:10.500 --> 00:03:14.460
interacting with those posts, so
that might not be too

00:03:14.500 --> 00:03:17.840
bad. And there is a reason why you can

00:03:17.980 --> 00:03:21.220
write more than you can read. You

00:03:21.320 --> 00:03:24.980
could think that you should be able to
read more than write because

00:03:25.000 --> 00:03:28.580
writes are more expensive to their
database, to their infrastructure

00:03:28.620 --> 00:03:32.360
it's the opposite.
They allow you to write more than they

00:03:32.400 --> 00:03:36.300
read. And just as a side note, X also

00:03:36.440 --> 00:03:39.620
has a new program which they're testing,
it's a pilot right

00:03:39.680 --> 00:03:43.520
now, uh, where they, um, want to give

00:03:43.580 --> 00:03:46.850
you, uh, pay per use access to their API.

00:03:46.860 --> 00:03:50.310
But it stays the same.
You have to pay to use it and it can get

00:03:50.320 --> 00:03:53.060
expensive. Now, why
are companies doing that?

00:03:53.100 --> 00:03:56.540
Well, the big answer, of course, is AI, or

00:03:56.900 --> 00:04:00.780
specifically, of course, gen AI. Because

00:04:00.860 --> 00:04:04.500
with the rise of gen AI, it has become

00:04:04.580 --> 00:04:08.560
clear that all that data
which these companies own, all

00:04:08.620 --> 00:04:12.040
these Reddit posts, all the posts on X,

00:04:12.680 --> 00:04:16.220
that is a valuable resource because

00:04:16.240 --> 00:04:19.420
those gen AI models, of

00:04:19.519 --> 00:04:23.460
course, need data in their training

00:04:23.500 --> 00:04:27.150
or for their training process. They,
data is the

00:04:27.300 --> 00:04:31.200
most important thing there because as we
all know, ChatGPT

00:04:31.260 --> 00:04:34.960
or the GPT models
were trained essentially on the entire

00:04:34.969 --> 00:04:38.840
data, publicly available data,
you could find on the internet.

00:04:39.000 --> 00:04:42.870
Um, and still,
these models will need vast amounts of

00:04:42.880 --> 00:04:46.560
data for their training. Nowadays,
of course, there is the entire concept or

00:04:46.680 --> 00:04:50.500
idea of using synthetic data as well as
real

00:04:50.540 --> 00:04:54.280
data for the training process,
and to my understanding, that

00:04:54.660 --> 00:04:58.630
seems to work quite well,
though we'll see if that maybe still is

00:04:58.660 --> 00:05:02.640
a problem and there is like a,
a ceiling due to the limited

00:05:02.680 --> 00:05:06.280
data that's available because the entire
data in the internet has already been

00:05:06.320 --> 00:05:09.740
consumed,
so now you're just generating more

00:05:09.780 --> 00:05:13.510
from that knowledge that
was gathered from that

00:05:13.540 --> 00:05:16.060
internet data,
so there might be a ceiling there.

00:05:16.080 --> 00:05:19.880
It's not entirely clear yet. Um,
but anyways,

00:05:19.900 --> 00:05:23.790
data is super important
and of course there's still new data being

00:05:23.860 --> 00:05:27.660
Now more data than ever
is generated by AI though, to be

00:05:27.680 --> 00:05:31.590
fair,
so that is synthetic data in the end,

00:05:31.680 --> 00:05:35.220
lots of data is being generated, uh,
including some data by humans

00:05:35.580 --> 00:05:39.460
on X and Reddit every day, and of course,

00:05:39.500 --> 00:05:43.380
those platforms don't want to give away
that data for free

00:05:43.390 --> 00:05:47.260
anymore.
They did in the past because we didn't see

00:05:47.320 --> 00:05:51.130
coming with, uh,
large language models and, um,

00:05:51.160 --> 00:05:54.880
now of course they want to protect their
data because a site

00:05:54.980 --> 00:05:58.720
like X of course sits on lots

00:05:59.160 --> 00:06:02.980
of data, lots of valuable posts,
at least to some degree,

00:06:03.020 --> 00:06:06.760
let's be honest.Most of the posts
are total BS, but at least

00:06:06.880 --> 00:06:10.740
some decent posts there
and definitely valuable in the sense

00:06:10.800 --> 00:06:14.780
of being valuable for training.
And those sites don't

00:06:14.800 --> 00:06:18.400
wanna give that data away for free
anymore, which is why

00:06:18.420 --> 00:06:22.400
they're locking it down. There also
is a reason why we

00:06:22.440 --> 00:06:25.210
see more and more web scraping

00:06:25.210 --> 00:06:29.180
businesses, uh,
coming up almost every day because now

00:06:29.200 --> 00:06:32.240
with large language models, even
if we ignore the training

00:06:32.340 --> 00:06:36.160
part, many of the applications
that we wanna build

00:06:36.220 --> 00:06:39.980
with help of large language models
or on top of large language models

00:06:40.100 --> 00:06:43.440
will need access to recent data.
If you're building a

00:06:43.450 --> 00:06:47.340
smart chatbot and you're using OpenAI's
models under the

00:06:47.360 --> 00:06:51.300
hood,
you probably wanna pull in some recent

00:06:51.380 --> 00:06:55.080
chatbot more useful.
You wanna add web search, you

00:06:55.160 --> 00:06:58.680
wanna be able to have your chatbot answer

00:06:58.740 --> 00:07:02.180
questions related to the most recent posts
on X.

00:07:02.220 --> 00:07:06.110
So you wanna pull
that data into your application and

00:07:06.160 --> 00:07:10.140
then reach your chat history
and the prompts you sent to

00:07:10.180 --> 00:07:14.120
the large language model with
that data that then hopefully

00:07:14.130 --> 00:07:17.560
allows the model to generate a better
answer, uh, for what the user

00:07:17.700 --> 00:07:21.540
asked. And that's why these sites are

00:07:21.640 --> 00:07:25.470
kind of locking down their APIs to make it
harder to get

00:07:25.580 --> 00:07:28.290
access to the data because in the past
they gave it away for free.

00:07:28.340 --> 00:07:32.240
They don't wanna do that anymore.
Obviously, there still are ways to

00:07:32.260 --> 00:07:36.020
get that data, as I mentioned.
There's a plethora of web

00:07:36.220 --> 00:07:39.980
crawling companies
and not all of these companies, uh,

00:07:40.000 --> 00:07:43.390
respect the fact
that certain sites don't want to get

00:07:43.540 --> 00:07:47.520
crawled.
Now I did actually a livestream on

00:07:47.580 --> 00:07:51.550
the topic of building our own web crawler,
um,

00:07:51.780 --> 00:07:55.420
a while back, and I'll, uh,
provide a link to that

00:07:55.880 --> 00:07:59.570
episode. You can watch the full, uh,
livestream episode, uh,

00:07:59.660 --> 00:08:03.220
below this episode, of course.
So you can build your own

00:08:03.260 --> 00:08:05.800
crawler and I did that with Crawl4AI.

00:08:05.820 --> 00:08:09.680
And essentially what you're building there
is, um, an

00:08:09.780 --> 00:08:12.600
application that spins up a browser and

00:08:13.040 --> 00:08:16.880
simulates being a user
and visiting a website to then

00:08:16.920 --> 00:08:20.460
extract that website content,
to extract the

00:08:20.500 --> 00:08:24.390
rendered HTML content and so on. Um,
that is

00:08:24.520 --> 00:08:27.760
how you can build
and use a crawler in the livestream.

00:08:27.980 --> 00:08:31.890
I just, um,
built it to crawl my own website,

00:08:31.940 --> 00:08:35.860
that clear. So I, uh,
did not start crawling X

00:08:35.900 --> 00:08:39.640
there because web scraping and crawling is

00:08:39.700 --> 00:08:43.539
kind of a gray zone. And, uh, there are

00:08:43.560 --> 00:08:47.220
many sites that clearly state in their
terms that they do

00:08:47.340 --> 00:08:51.330
not allow web crawling. So you
are violating those terms if you

00:08:51.460 --> 00:08:54.840
do, which is why sites like Firecrawl, for

00:08:54.900 --> 00:08:58.660
example, won't crawl X links. If I

00:08:59.180 --> 00:09:03.060
take an X link and I try to scrape

00:09:03.100 --> 00:09:06.860
that, uh, I'll get an error that this
is not

00:09:06.920 --> 00:09:10.900
supported.
Or actually here it doesn't even start as

00:09:10.940 --> 00:09:13.000
it seems in the past, I did get an error.

00:09:13.520 --> 00:09:15.710
So there are sites that don't allow that.

00:09:15.740 --> 00:09:19.450
There probably also are sites that do,
and you can

00:09:19.480 --> 00:09:22.670
definitely build your own crawler
that doesn't give

00:09:23.780 --> 00:09:27.560
anything about anything
and extract any content of any

00:09:27.660 --> 00:09:31.280
site you wanna extract. Now I will say,
of course, that many

00:09:31.360 --> 00:09:35.100
sites also try to implement some technical
hurdles that make it

00:09:35.160 --> 00:09:38.930
harder to crawl them, but in the end
if you really want to, you can

00:09:38.980 --> 00:09:41.210
get around pretty much all of them.

00:09:41.260 --> 00:09:45.140
It might not be legal,
it might be violating their terms, but

00:09:45.180 --> 00:09:49.130
it is possible. Because, and
that takes us back

00:09:49.160 --> 00:09:53.000
to the beginning, to the main topic,
because of course all that data

00:09:53.200 --> 00:09:57.040
is super valuable. However, this does have

00:09:57.420 --> 00:10:01.080
I believe a real downside or a, an

00:10:01.140 --> 00:10:04.480
implication that's not great for us as

00:10:04.490 --> 00:10:08.400
developers. Because I totally get
that these sites don't wanna give

00:10:08.460 --> 00:10:11.830
away access to their data,
and just as a side note,

00:10:11.860 --> 00:10:15.840
it's kind of not their data.
It's the data of the users using the

00:10:15.880 --> 00:10:18.220
site, but that's a whole different story.

00:10:18.240 --> 00:10:21.650
But I get that they don't wanna give away
access to this data.

00:10:21.680 --> 00:10:25.350
The problem of course is
that kind of as a, a,

00:10:25.740 --> 00:10:29.590
an additional casualty,
we as developers are limited

00:10:29.620 --> 00:10:32.470
in what we can build, uh, with those APIs.

00:10:32.470 --> 00:10:36.340
Sure,
you might get approval for the Reddit API

00:10:36.380 --> 00:10:40.260
something they're happy with.
But of course you also might

00:10:40.340 --> 00:10:44.260
not get that approval. And for X you have

00:10:44.400 --> 00:10:47.880
to pay quite a bit of money depending on
what you're

00:10:47.920 --> 00:10:51.500
building,
even if what you're building has nothing

00:10:51.540 --> 00:10:55.280
extracting that data
and using it for model

00:10:55.360 --> 00:10:58.940
training or anything like that.
It limits the

00:10:59.020 --> 00:11:02.460
amount of useful stuff we can build on top
of other

00:11:02.560 --> 00:11:05.980
services and sites. And
that of course in turn, um,

00:11:06.840 --> 00:11:10.580
might also hurt those sites because some
useful products from which they might

00:11:10.620 --> 00:11:13.860
benefit then maybe won't get built.

00:11:13.880 --> 00:11:17.160
But of course I guess that
is a price they're happy to pay

00:11:17.220 --> 00:11:21.040
because either they'll get paid by these

00:11:21.080 --> 00:11:24.750
people that build products on top of them,
or at least they prevent that

00:11:24.780 --> 00:11:28.700
data extraction. So yeah, uh, I expect

00:11:28.710 --> 00:11:32.370
that we will see more sites and

00:11:32.480 --> 00:11:35.880
services, uh, locking down their APIs.

00:11:36.020 --> 00:11:39.290
I think we'll see more sites becoming
pretty

00:11:39.319 --> 00:11:42.960
protective about their data,
pretty aggressive

00:11:42.970 --> 00:11:46.400
against crawlers, which
is absolutely their right,

00:11:46.440 --> 00:11:50.310
there. Um, but I think
that might also hurt

00:11:50.500 --> 00:11:54.319
us as developer because it kind of limits
the, the stuff we

00:11:54.360 --> 00:11:57.540
can build around other popular services
and

00:11:57.640 --> 00:12:01.220
sites. These are my two cents.
What do you think about this

00:12:01.300 --> 00:12:02.100
topic?