It is the same stream object being passed to multiple crawlers via userData. Wasn't sure how else to pass the stream object to the routers. I'm now trying something like this...
\nlet _router: RouterHandler<CheerioCrawlingContext<any, any>>;\n\nexport const initRouter = (writer: WritableStreamDefaultWriter<string>) => {\n if (_router== undefined) {\n _router= createCheerioRouter();\n }\n\n // Metro, Food Basics\n _router.addHandler('DETAIL', async ({ request, $, log, pushData }) => {\n ...\n\n if (writer) {\n await writer.write(JSON.stringify(product));\n }\n });\n\n\n // This is a fallback route which will handle the start URL\n // as well as the LIST labeled URLs.\n _router.addDefaultHandler(async ({ request, enqueueLinks, log }) => {\n ...\n });\n\n return _router;\n}UPDATE: Tried it and this is working for me
","upvoteCount":0,"url":"https://github.com/apify/crawlee/discussions/2565#discussioncomment-9947894"}}}
-
|
To give a little context I am trying to stream my scraped data back to the user via API endpoint. I'm running into an issue where I pass the WriteableStream to the userData and within the route handlers I am writing the data to the stream. The issue is the userData does not maintain the properties and functions of my object being passed in, "ERROR CheerioCrawler: Request failed and reached maximum retries. TypeError: stream.write is not a function" const { readable, writable } = new TransformStream();
const writer = writable.getWriter();
async function startCluster(writer: WritableStreamDefaultWriter<string>) {
const crawler = new CheerioCrawler(
Object.assign(
{ requestQueue: await RequestQueue.open(CRAWLERS.UNIQUE_KEY) },
crawlOptions
),
new Configuration({
defaultKeyValueStoreId: CRAWLERS.UNIQUE_KEY,
defaultDatasetId: CRAWLERS.UNIQUE_KEY,
defaultRequestQueueId: CRAWLERS.UNIQUE_KEY
}));
const crawl1 = crawler.run([
{
url: `https://www.abc.com`,
userData: {
stream: writer
}
},
]);
await crawl1;
}
export const router = createCheerioRouter();
router.addDefaultHandler(async ({ request, enqueueLinks, log }) => {
await enqueueLinks({
selector: 'details-link',
label: 'DETAIL',
userData: request.userData
});
});
router.addHandler('DETAIL', async ({ request, $, log, pushData }) => {
...
await pushData(product);
const stream = request?.userData?.stream as WritableStreamDefaultWriter<string>;
if (stream) {
await stream.write(JSON.stringify(product)); // <---- This is where I run into a problem.
//"ERROR CheerioCrawler: Request failed and reached maximum retries. TypeError: stream.write is not a function"
}
});
startCrawler(writer); |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 7 replies
-
|
Maybe have a map of request.uniqueKey to your stream handles instead of adding it there? |
Beta Was this translation helpful? Give feedback.
-
|
I don't have time to write you a complete example, please show me what you tried first. |
Beta Was this translation helpful? Give feedback.
-
|
Uh sorry, I see you already have the example code in the OP :] |
Beta Was this translation helpful? Give feedback.
-
|
Hmm looking at your code I am honestly not sure why are you using adding the stream to the user data? You got one stream per crawler instance, right? |
Beta Was this translation helpful? Give feedback.
-
|
It is the same stream object being passed to multiple crawlers via userData. Wasn't sure how else to pass the stream object to the routers. I'm now trying something like this... let _router: RouterHandler<CheerioCrawlingContext<any, any>>;
export const initRouter = (writer: WritableStreamDefaultWriter<string>) => {
if (_router== undefined) {
_router= createCheerioRouter();
}
// Metro, Food Basics
_router.addHandler('DETAIL', async ({ request, $, log, pushData }) => {
...
if (writer) {
await writer.write(JSON.stringify(product));
}
});
// This is a fallback route which will handle the start URL
// as well as the LIST labeled URLs.
_router.addDefaultHandler(async ({ request, enqueueLinks, log }) => {
...
});
return _router;
}UPDATE: Tried it and this is working for me |
Beta Was this translation helpful? Give feedback.
It is the same stream object being passed to multiple crawlers via userData. Wasn't sure how else to pass the stream object to the routers. I'm now trying something like this...